fip mapper for new standard #376

dougollerenshaw · 2025-10-16T02:39:58Z

Implements a FIP (Fiber Photometry) mapper that transforms intermediate metadata from the extractor into schema-compliant Acquisition objects.
Follows the SmartSPIM mapper pattern (transform-only, no I/O). Creates 3 channels per fiber (Green, Isosbestic, Red) reflecting the temporal multiplexing setup. Integrates with metadata service endpoints for intended measurements and procedures data to validate which fibers were actually implanted.
External API calls are mocked in tests so they run fast (~1.5s) and work in CI/CD. Falls back to ROI-based inference if service endpoints are unavailable.
TODO:
Camera exposure times (currently using -1 as placeholder)
Filter specifications
Full connection graph
All tests pass. Example script runs successfully.

Closes: https://github.com/AllenNeuralDynamics/aind-physio-arch/issues/811

- Implements FIPMapper class for transforming intermediate FIP metadata to Acquisition schema - Follows SmartSPIM mapper pattern: extract → transform → save - Maps FIP hardware configs (LEDs, cameras, patch cords, fibers) to schema objects - Zero-indexed naming: ROI N → Patch Cord N → Fiber N (implanted) - Includes comprehensive TODO documentation for future enhancements - Adds unit tests with 100% passing coverage - Includes working example script and sample acquisition.json output Key features: - Transforms session metadata (times, experimenter, subject, rig) - Builds Channel objects with detectors and emission wavelengths - Creates PatchCordConfig for each ROI/fiber pair - Includes implanted fiber identifiers in active devices - Handles timezone-aware timestamps - Validates against aind-data-schema v2.0+ TODO items documented for future work: - Connection graph (requires instrument metadata endpoint) - Intended measurements (requires ROI mapping) - Filter specifications (requires rig config enhancement) - LED-to-ROI mapping (requires rig config enhancement)

…data Follow standard AIND mapper pattern: - Add run_job() method for one-step transform + write workflow - Make output filename configurable at initialization (defaults to 'acquisition.json') - Update write() to return Path and use mapper's output_filename attribute - Example uses custom filename 'fip_example_acquisition.json' to avoid conflicts Document FIP system architecture in detail: - Each implanted fiber has 3 temporal-multiplexed channels (60Hz cycling) - Green channel: 470nm excitation → ~520nm emission → Green CMOS - Isosbestic channel: 415nm excitation → ~520nm emission → Green CMOS (same camera!) - Red channel: 565nm excitation → ~590nm emission → Red CMOS - Clarify current implementation creates 1-2 channels per fiber vs needed 3 channels - Document missing pieces: LED-to-ROI mapping, ROI-to-measurement mapping Add wavelength inference and metadata: - New _get_led_wavelength() maps LED names to excitation wavelengths (415nm, 470nm, 565nm) - Update _infer_emission_wavelength() with accurate values (520nm green/iso, 590nm red) - Include wavelengths in LED device names (LED_UV_415nm, LED_BLUE_470nm, LED_LIME_565nm) - Update emission wavelength from 525nm/600nm to 520nm/590nm based on system specs Update documentation and examples: - README shows both run_job() and transform()+write() patterns - Note that output filename is configurable but defaults to standard 'acquisition.json' - Example script demonstrates custom filename usage - All tests pass (12/12) This commit maintains backward compatibility while aligning with AIND mapper standards and providing better documentation for future full 3-channel-per-fiber implementation.

…m metadata service - Add requests import for HTTP calls to metadata service - Implement get_intended_measurements() method to query intended_measurements endpoint - Method queries http://aind-metadata-service/intended_measurements/{subject_id} - Returns dictionary mapping Fiber_N to channel measurements (R, G, B, Iso) - Handles HTTP errors gracefully with warnings - Tested successfully with subject 726649 Next step: Integrate into channel creation logic to populate intended_measurement field

Changes: - Call get_intended_measurements() in _transform() using subject_id - Pass intended_measurements to _build_configurations() - Add _get_channel_measurement() helper to map ROI+camera to measurement - Populate channel intended_measurement field from metadata service - Maps camera types: red camera→R channel, green camera→G channel - Handles missing measurements gracefully (returns None) Documentation: - Add CURRENT IMPLEMENTATION section showing what's done - Clean up TODO list to only show remaining work - Remove completed items: intended_measurement, fiber IDs, LED wavelengths - Focus TODO on: 3-channel structure, filters, connection graph, detailed device metadata Successfully tested with subject 726649 (calcium, dopamine, control measurements).

Major changes: - Now creates 3 channels per fiber: Green, Isosbestic, and Red - Green and Isosbestic channels share same detector (green CMOS) but have different excitation wavelengths - Each channel gets its own intended_measurement from metadata service (G, Iso, R) - Channel names: Fiber_N_Green, Fiber_N_Isosbestic, Fiber_N_Red Implementation details: - Simplified patch cord creation logic to iterate over fibers, not cameras - Each fiber gets all 3 channels if cameras are present - Removed _get_channel_measurement() helper (no longer needed) - Updated documentation to reflect 3-channel implementation Testing: - All 12 unit tests pass - Successfully creates 3 channels per patch cord - Example output verified with 4 fibers × 3 channels each = 12 total channels This completes the channel structure implementation!

… physics Changes: - Define module-level constants for LED excitation wavelengths: * EXCITATION_UV = 415nm (UV LED) * EXCITATION_BLUE = 470nm (Blue LED) * EXCITATION_YELLOW = 565nm (Yellow/Lime LED) - Define emission wavelength constants: * EMISSION_GREEN = 520nm (Green camera) * EMISSION_RED = 590nm (Red camera) - Clarify excitation → emission physics throughout: * Blue LED (470nm) → green emission (520nm) [GFP signal] * UV LED (415nm) → green emission (520nm) [isosbestic control] * Yellow LED (565nm) → red emission (590nm) [RFP signal] - Rename variables to reflect LED type (blue_led, uv_led, yellow_led) - Update all docstrings to clearly explain excitation/emission relationship - Replace all magic number references with named constants Testing: - All 12 unit tests pass - No functional changes, purely refactoring for code clarity

Changes to mapper: - Add get_implanted_fibers() method to fetch from procedures endpoint * Queries http://aind-metadata-service-dev/api/v2/procedures/{subject_id} * Parses Brain injection procedures to identify implanted fiber indices * Maps relative_position (Right→0, Left→1) to fiber indices * Timeout set to 60s due to slow endpoint (~30s response time) - Update _build_configurations() to accept implanted_fibers parameter * Only creates PatchCordConfig objects for actually implanted fibers * Falls back to ROI-based inference if procedures data unavailable * Prevents creating connections to non-existent implanted fibers - Update _get_active_devices() to accept implanted_fibers parameter * Only includes patch cords and fibers that actually exist * Consistent with patch cord configuration logic - Update mapper documentation to reflect procedures integration Changes to tests: - Mock get_implanted_fibers() to return [0, 1] (two fibers) - Mock get_intended_measurements() to return sample measurements - Add setUp/tearDown methods to start/stop mocking - Tests now run in ~1.3s instead of ~124s (100x faster!) - Tests will work in GitHub CI/CD (no external service dependencies) Testing: - All 12 unit tests pass - Falls back gracefully if procedures endpoint fails - Maintains backwards compatibility with ROI-based fiber detection This addresses the requirement to validate which fibers are actually implanted using surgical procedure records, while keeping tests fast and portable.

Changes: - Add PLACEHOLDER_CAMERA_EXPOSURE_TIME = -1 constant at module level * Clearly indicates missing data (impossible negative exposure time) * Easy to find and replace when actual camera metadata is available * Well-documented with TODO comment - Use constant in all three channel DetectorConfig objects: * Green channel (Blue LED → Green CMOS) * Isosbestic channel (UV LED → Green CMOS) * Red channel (Yellow LED → Red CMOS) - Remove standalone DetectorConfig objects from configurations list * Now only created inline within Channel objects * Prevents duplication and inconsistency - Update test to check for DetectorConfig within PatchCordConfig channels * Verifies proper nesting structure - Update TODO documentation to reference the placeholder constant Testing: - All 12 unit tests pass - Exposure time clearly marked as placeholder in code - No fake/hallucinated values pretending to be real data The -1 value makes it immediately obvious to anyone inspecting the acquisition.json that this is placeholder data requiring replacement.

Changes: - Corrected logic to look for 'Probe implant' procedures (not 'Brain injection') - Check for implanted_device with object_type == 'Fiber probe' - Extract fiber index from device name (e.g., 'Fiber_0' -> 0, 'Fiber_1' -> 1) - Fixed indentation issues in _build_configurations fallback logic This addresses the user's feedback that the previous implementation was incorrectly mapping brain injection relative_position (Left/Right) to fiber indices. The correct approach is to parse the actual Fiber probe implant procedures and extract the fiber indices directly from the device names. Testing: - All 12 unit tests pass - Mock still returns [0, 1] for test consistency - Real procedures endpoint would return indices like [0, 1, 2] based on actual 'Fiber_0', 'Fiber_1', 'Fiber_2' device names

Ahad-Allen

A few questions, but mostly look fine to me

Ahad-Allen · 2025-10-16T07:05:11Z