Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
|
|
|
import os
|
|
|
|
|
|
2026-06-04 20:30:04 -04:00
|
|
|
from sqlalchemy import MetaData, event, text
|
|
|
|
|
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
|
|
|
from sqlalchemy.pool import NullPool
|
2026-06-04 20:30:04 -04:00
|
|
|
|
2026-07-07 17:50:51 -04:00
|
|
|
from config import DATABASE_URL
|
2026-06-04 20:30:04 -04:00
|
|
|
|
Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
|
|
|
# Pooled connections are bound to the event loop they were created on. Long-lived
|
|
|
|
|
# servers (uvicorn, the ingester) want a pool; short-lived / test processes that
|
|
|
|
|
# open a fresh event loop per unit (asyncio.run, pytest) must not reuse pooled
|
|
|
|
|
# connections across loops, so allow a NullPool (new connection per session).
|
|
|
|
|
_NULL_POOL = os.getenv("DB_NULL_POOL", "").lower() in ("1", "true", "yes")
|
|
|
|
|
|
2026-06-04 20:30:04 -04:00
|
|
|
engine = create_async_engine(
|
Add NASA FIRMS active-fire ingest + /api/fires; API keys management page
Coherent merge of two coordinated features on the shared working tree:
FIRMS fire heatmap (backend, t_6e404c14):
- app/fire_sources.py: fetch FIRMS VIIRS area CSV (free MAP_KEY) -> NATS events.fire
- fires hypertable (TimescaleDB, 1-day chunks) with natural-key PK
(latitude, longitude, acq_time, satellite); idempotent ON CONFLICT DO NOTHING
- alembic/versions/002_fires.py; GET /api/fires?bbox=&since= (JSON only)
- POST /api/ingest/fires; ~15 min poll loop (FIRMS_INTERVAL=900) in ingester
- env-driven config (FIRMS_MAP_KEY/DATASET/BBOX/INTERVAL); docs/firms.md covers
the zero-cost GIBS VIIRS_SNPP_Thermal_Anomalies_375m_All tile alternative
- 18 tests (parser, mapping, idempotency, API contract) verified vs real
TimescaleDB+PostGIS (localhost/osint-dashboard-pg image)
API keys page (frontend, t_4433cff2):
- app/keystore.py: api_keys table (self-creating), FIRMS/GEMINI/TELEGRAM
registry with format validation, ****last4 masking, get_api_key()
- GET/POST/DELETE /api/keys (never returns full values); Keys tab in index.html
DB_NULL_POOL env switch in app/database.py enables a NullPool for tests /
short-lived processes that open a fresh event loop per unit.
2026-08-24 15:37:42 -04:00
|
|
|
DATABASE_URL,
|
|
|
|
|
echo=False,
|
|
|
|
|
**({"poolclass": NullPool} if _NULL_POOL else
|
|
|
|
|
{"pool_size": 5, "max_overflow": 10, "pool_recycle": 300}),
|
2026-06-04 20:30:04 -04:00
|
|
|
)
|
|
|
|
|
async_session = async_sessionmaker(
|
|
|
|
|
engine, class_=AsyncSession, expire_on_commit=False
|
|
|
|
|
)
|
|
|
|
|
metadata = MetaData()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
async def init_extensions():
|
|
|
|
|
"""Initialize PostGIS and TimescaleDB extensions on first connection."""
|
|
|
|
|
async with engine.connect() as conn:
|
|
|
|
|
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS postgis"))
|
|
|
|
|
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS timescaledb"))
|
|
|
|
|
await conn.commit()
|