SpikeInterface 0.101.0 release notes

19th July 2024

Main changes:

• Implementation of SortingAnalyzer (#2398)

• Improved auto-merging functions and enable SortingAnalyzer to merge units and extensions (#3043, #3154, #3203, #3208)

• Added framework for hybrid recording generation (#2436, #2769, #2857)

• Refactored motion correction with the Motion class and the DREDGE AP and LFP methods (#2915, #3062)

• Extendeded benchmarking of sortingcomponents (#2501, #2518, #2586, #2811, #2959)

• Added a powerful drift generator module (#2683)

core:

• Implement a simple system to have backward compatibility for Analyzer extension (#3215)

• Units aggregation preserve unit ids of aggregated sorters (#3180)

• Better error message for BaseExtractor.load() (#3170)

• Saving provenance with paths relative to folder (#3165)

• Fix segment start/end frame None in concatenate_recordings (#3161)

• Implement apply_merges_to_sorting() (#3154)

• More explicit consistency checks for Channel aggregation (#3151)

• Fix serializability in save function and InjectDriftingTemplatesRecording (#3130)

• Fix t_starts not propagated to save_to_memory (#3120)

• Round instead of int for time_to_sample_index (#3119)

• Add time vector case to get_durations (#3118)

• Reordering recording, sorting args (#3094)

• extractor_dict_iterator for solving path detection in object kwargs (#3089)

• Check start_frame/end_frame in BaseRecording.get_traces() (#3059)

• Relax numpy (#3056)

• Add scale_to_uV preprocessing (#3053)

• Add more helpful error when a non-existent id is passed to extractors (#3052)

• Propagate FrameSlice default behavior to frame_slice and time_slice (#3051)

• fix spike_vector_to_indices() (#3048)

• Skip loading info.json into ext.data (#3047)

• Proposal to implement a sorting_analyzer.merge_units() syntax (#3043)

• add recording.time_slice like recording.frame_slice (#3034)

• Numpy 2.0 cap Fix most egregorious deprecated behavior and cap version (#3032)

• Avoid redirection to not implemented method in BaseRecording.select_channels (#3002)

• Add documentation to frame_slice in recording and drop redundant inheritance (#3001)

• Add select chanel and select unit method for template objects (#2989)

• Missed has_scaled (#2979)

• Fixed select_channels (#2977)

• Make _set_probes private (#2949)

• Add limit for n_jobs with a max of os.cpu_count() (#2940)

• API: switch has_scaled -> has_scaleable_traces (#2924)

• Making chunk_size=None behavior for n_jobs=1 explicit (#2902)

• Fix remaining verbose clashing (#2901)

• Remove verbose from job_kwargs (#2898)

• Put get_extension (to trigger auto-import) before get_children_dependencies for deleting Analyzer extensions to prevent KeyError (#2888)

• Add peak_to_peak mode to get_templates_amplitude (#2885)

• Protect zarr properties of type object (#2884)

• Avoid returning std when asking for only mean/average with templates extension (#2868)

• Add binary suffixes to job tools memory specification (e.g. 1GiB vs 1GB) (#2858)

• Possible fix for #2837 (#2854)

• Load is_scaled in from zarr Template representation (#2852)

• Use the is_scaled attribute inside of template_tools functions (#2848)

• Add extra check in fix_job_kwargs (#2829)

• Zarr: extract time vector once and for all! (#2828)

• Add html representation for recording objects (#2803)

• Remove SharedMemory from SortingAnalyzer and use NumpySorting instead. (#2801)

• Channel aggregation preserves names (#2797)

• Only assign memmap within boundaries for write_binary (#2796)

• Improve verbose of ChunkRecordingExecutor (#2795)

• Add select_channels method to base recording (#2786)

• Analyzer set recording (#2785)

• Add name as an extractor attribute for __repr__ purposes (#2784)

• Compute quality metrics after pipeline nodes (#2773)

• Check for SortingAnalyzer return_scaled in template_tools (#2752)

• Template amplitude should not be returned as absolute value (#2750)

• Fix performance issue for aggregate_channels (#2736)

• Multiple verbose arguments fix (#2728)

• Fix pipelining of extensions (#2715)

• Seepd up random_spikes_selection() and put “all” method in it (#2700)

• Extend compute for extension kwargs and pipeline nodes (#2693)

• Add ‘all’ method for random_spikes (#2691)

• Move “return_scaled” options global to SortingAnalyzer. (#2684)

• Extend zarr compression options (#2643)

• Fast templates: compute std with and extend to widgets (#2623)

• SortingAnalyzer: get_extension_default_params (#2616)

• Load pca from waveform extractor and waveform from Zarr (#2613)

• Seeding silence periods (#2611)

• Implement get_parent() function (#2610)

• Fix analyzer sampling frequency check (#2606)

• Proposal to inject anisotropy while creating fake templates (#2600)

• Fix for sorting from_peaks with lexsort (handling multi segments) (#2594)

• Save extension class info. (#2585)

• Fix to impose sorting of the peaks in from_peaks() (#2584)

• Extension delete on recompute (#2579)

• Proposal for auto import extensions module. (#2571)

• Add check_json to zarr for SortingAnalyzer for sorting_provenance file writing (#2559)

• Add more backwards compatibly for MockWaveformExtractor (#2546)

• Make binary recording memmap efficient III (Avoiding spikes by only reserving memory equal to the trace) (#1781)

• Implementation of SortingAnalyzer (#2398)

preprocessing:

• Update doc handle drift + better preset (#3232)

• Remove name class attribute in preprocessing module (#3200)

• Add option to use ref_channel_ids in global common reference (#3139)

• Adding option to overwrite while doing correct_motion and saving to a folder (#3088)

• Docstrings preprocessing update, fix PR01 and PR02 (#3016)

• Fix deepinterpolation tests (#2971)

• Improve phase shift memory efficiency (#2946)

• Fix highpass-spatial-filter return dtype (#2864)

• Update detect_bad_channels docs (#2783)

• Regularize whitening (#2744)

• Seeding silence periods (#2611)

• Added round option to recording.astype (#2513)

• Small bug fix in zscore : gain / offset must be reset (#2510)

extractors:

• Fix naming when all_annotations=True in neo base extractors (#3198)

• Propagate option for using neo channel names as ids in all neo extractors without changing the deafult behavior (#3195)

• Update nwbextractors docstring (#3179)

• Build extractor dicts automatically (#3169)

• Remove unused legacy class name/mode from extractors (#3153)

• Improved conditions for closing nwb when using hdf5 backend (#3150)

• Use names as channel ids in plexon2 (#3065)

• Fix intan kwargs (#3054)

• Remove scipy upper bound for streaming extractors (#3023)

• Fix timestamps access in NWB extractor (#2993)

• Mcsh5 offsets and proper scaling in uV for return_scaled (#2988)

• Fix bug with nan values (#2970)

• Add argument to IntanRecordingExtractor for opening files with discontinous timestamps (#2969)

• Add extractors for SiNAPS Research Platform (#2952)

• Add zarr.open kwargs to fetch_available_electrical_series_paths in NWBRecordingExtractor (#2909)

• Undefined variable IBL extractor (#2906)

• Remove unused argument in nwb extractors read_file_from_backend (#2904)

• Add static method to NWBRecordingExtractor to fetch the available electrical series (#2903)

• Remove unused extractor attribute extractor_name (#2874)

• Avoid duplicating channel names as property in neo base extractor when all_annotations=True (#2845)

• Fix EID in IBL (#2714)

• Deprecate ros3 as a streaming method (#2662)

• Fix channel gains in NwbRecordingExtractor with backend (#2661)

• Open Ephys: Use discovered recording ids to load sync timestamps (#2655)

• ADD: Neuropixels ProbeGroup to SpikeGadgetsRecordingExtractor. (#2641)

• Add extra_requirements for nwb extractors (#2637)

• Remove lazy typing in nwb (#2635)

• Add supportfor IntanRecordingExtractor “one-file-per-signal” and “one-file-per-channel” formats (#2630)

• IBL extractors revived (#2617)

• Add strict_gap_mode in read_neuralynx to reflect neo. (#2550)

• Use has_channel_location instead of get_property(“contact_vector”) (#2475)

• Implementing read_spikeglx_event() (#2442)

sorters:

• Patch for SC2 after release AND bugs in auto merge (#3213)

• Improve error log to json in run_sorter (#3057)

• Add support for kilosort>=4.0.12 (#3055)

• Make sure we check is_filtered() rather than bound method during run basesorter (#3037)

• Add more container dependency checks in run_sorter (#3024)

• whiteningRange added as Kilosort 2.5 parameter (#2997)

• Patch for regularizing folder arguments (#2968)

• Regularize folder argument for user in run_sorter options (#2917)

• Update tridesclous2 (#2860)

• Benchmark sorting components + Tridesclous2 improvement (#2811)

• Update KS4 parameters (#2810)

• Add parameter for irc1 (#2809)

• Fix KS4 for v>=4.0.5 and simplify skip correction (#2774)

• Improvements for circus 2 (#2768)

• Remove separate default job_kwarg n_jobs for sorters (#2712)

• Update ks4 wrapper for v4.0.3 (#2701)

• Improve caching of MS5 sorter (#2690)

• Add whitening before peeler TDC2 SC2 (#2679)

• add option to kilosort4 to force cpu even if cuda is available when small gpu (#2647)

• Add batch size to KS4 wrapper (#2592)

• Fix for template matching in circus (#2583)

• Use GPU in Kilosort4Sorter (#2577)

• Circus2 improvements (#2574)

• Simple sorter better params (#2547)

• Kilosort4 Wrapper (#2529)

• Fix singularity issue in deleting py_user_folder (#2493)

• Tridesclous2 update (#2267)

postprocessing:

• Fix pca transform error (#3178, #3224)

• Fix spike_vector_to_indices() (#3048)

• Remove un-used argument (#3021)

• Optimize numba cross-correlation and extend correlograms.py docstrings and tests (#3017)

• Moving unit_localization into unit_locations for consistency (#2992)

• Template similarity lags (#2941)

• Fix isi_histograms when using numba (#2632)

• Revert change in set_params for unit locations (#2624)

qualitymetrics:

• Fix nn pca_metric computation and update tests (#3138)

• Add peak_to_peak mode to SNR (#3009)

• Add test to check unit structure in quality metric calculator output (#2973)

• Fix math error in sd_ratio (#2964)

• Remove “waveforms” dependency for compute_quality_metrics() (#2707)

• compute_synchrony_metrics update (#2605)

curation:

• Add apply_curation() (#3208)

• Port auto-merge changes and refactor (#3203)

• Implement apply_merges_to_sorting() (#3154)

• Proposal of format to hold the manual curation information (#2933)

• Provide useful message if sorting used instead of analyzer for remove_duplicate_units (#2790)

• Fix split in more than 2 units and extend curation docs and tests (#2775)

• Enhancing curation : get_potential_auto_merge() (#2753)

• Make sure has_exceeding_spikes also checks for negative spikes. (#2727)

• Allow for remove_excess_spikes to remove negative spike times (#2716)

• Add function in addition to class option for MergeUnitsSorting, SplitUnitSorting, and CurationSorting (#2659)

• Allow auto_merge to work with fast_templates (#2551)

widgets:

• Channel ids sortingview (#3205)

• Since the gui now has curation we should allow users to set curation when making sorting summary (#3104)

• Add plot_drift_raster_map widget (#3068)

• Fix bug in plot_potential_merges (#3044)

• Add peaks_on_probe widget. (#3022)

• Plot potential merges (#3003)

• Fix the new way of handling cmap in matpltolib. This fix the matplotib 3.9 problem related to this. (#2891)

• Extend plot waveforms/templates to Templates object (#2856)

• Fix bug in plot templates (#2850)

• Add errors to ensure functions in Base of widgets (#2791)

• Add color_channels option in in plot_probe_map (#2740)

• Extend capabilities of plot_traces and spikes_on_traces (#2737)

• Enable waveform-less plot templates (#2692)

• Fix depth location in spikes on traces map (#2676)

• Improve spikes on traces (#2646)

generation:

• Remove unit_params_range from generate.py (#3121)

• Out-of-place multiply needed when using amplitude scaling in the drifty hybrid recording (#3115)

• Update tests against template library (#3000)

• Add select channel and select unit method for template objects (#2989)

• Template fetching methods (#2857)

• Load is_scaled in from zarr Template representation (#2852)

• Add is_scaled to Templates object (#2842)

• Add template generate_recording_from_template_database (#2769)

• Add a missing seed in generator (#2706)

• Generate drift (#2683)

• Changing the generation of fake template with a decaying exponential and fix torch locally exclusive (#2674)

• Proposal to inject anisotropy while creating fake templates (#2600)

• Fix scaling of inject templates with copy (#2521)

• Tools for Generation of Hybrid recordings (#2436)

• Remove default values used as expressions in generate.py. (#2345)

sortingcomponents:

• Fix estimate_motion when time_vector is set (#3218)

• Fix select peaks (#3132)

• Dredge lfp and dredge ap (#3062)

• Use “available” for memory caching (#3008)

• get_spike_prototype can have NaN sometimes (#2980)

• Benchmarks components: plotting utils (#2959)

• Internal motion API, aka Motion object (#2915)

• Components of SC2 (#2870)

• Benchmark sorting components + Tridesclous2 improvement (#2811)

• Change “idw” to “kriging” in spatial_interpolation_kernel parameters (#2708)

• Benchmarks for sorting components (#2586)

• Fix for template matching in circus (#2583)

• Circus2 improvements (#2574)

• Matched filtering to enhance peak detection (#2259)

• Benchmark matching (#2518)

• Initial benchmark components refactor (#2502)

• Remove WaveformExtractor from Templates matching (#2485)

documentation:

• Analyzer docstring cleanup (#3220)

• Eradicate sphinx warnings (#3188)

• Convert doc references from wf_extractor -> sorting_analyzer (#3185)

• Add explainer of compute always computing in the analyzer (vs WaveformExtractor behavior) documentation (#3173)

• Extend docs and API for generation module (#3167)

• analyse -> analyze in Neuropixels docs (#3143)

• Add pooch to docs extra (#3124)

• Docstrings extractors update, fix PR01 and PR02 #3016 (#3076)

• Unify compute_isi_violation docs and add UltraMegaSort2000 citation (#3070)

• Correct docs and docstrings for compute_template_metric units (#3066)

• Add bool type hint to functions in core module (#3064)

• Fix the probe handling tutorial (#3063)

• Add more helpful error when a non-existent id is passed to extractors (#3052)

• Docstrings preprocessing update, fix PR01 and PR02 (#3016)

• Add documentation to frame_slice in recording and drop redundant inheritance (#3001)

• Add a jQuery extension to enable search (#2962)

• Standardize colon spacing in docstrings (#2950)

• Make a how to loading data into a Sorting manually (#2944)

• Add references to documentation page (#2938)

• Standardise qualitymetrics docstrings to numpydocs standard (#2935)

• Add _params_descriptions for simple sorter (#2894)

• Extend docstrings for amplitude scaling and collisions (#2893)

• Improve save_to_folder docstring (#2873)

• Unpin sphinx and add networkx dependency. (#2861)

• Switch Visualise to Visualize (#2830)

• Add typing to write_binary_recording (#2794)

• Update detect_bad_channels docs (#2783)

• Reorganising documentation into Getting Started, Tutorial and How To (#2778)

• Switch exporters docs to SortingAnalyzer (#2762)

• Update amplitude cutoff documentation (#2756)

• Update SortingAnalyzer docs (see #2711) (#2719)

• Add basic repr to Templates object (#2718)

• Fix example broken by #2684 (#2717)

• Add a “how to” for combining recordings (#2713)

• Fix SortingAnalyzer tutorial (#2648)

• Sync documentation with bug fixes (#2645)

• WaveformExtractor -> SortingAnalyzer doc page (#2633)

• Update get_started.rst with info for reloading saved binaries (#2622)

• Analyzer core extension: improve docstrings (#2608)

• Improve documentation for excess spikes error in FrameSliceSorting. (#2604)

• Put SortingAnalyzer in get started (#2582)

• Clarify status of main and Sortinganalyzer (#2580)

• Add note about recommendation not to change batch size in Kilosort1-3 (#2564)

• SortingAnalyzer core documentation (#2563)

• Cleanup of modules gallery docs (post SorterAnalyzer) (#2552)

• Add note to explain that Command Prompt is currently the only officially supported shell for Windows (#2494)

• Add documentation of preprocessing and sorting split by channel group. (#2316)

• SortingAnalyzer in “how to” analyze neuropixels (#2520)

continuous integration:

• Add test_extractors to full test with codecov (#3141)

• Add plexon2 tests action for install wine and drop full (linux only) tests (#3101)

• Add machinery to run test only on changed files (#3084)

• Remove cached dependencies (#3080)

• Add macos and windows to cache cron jobs (#3075)

• Update cron job for installation_tips (#3038)

• Improve profile imports test (#2955)

• Modify CI for new dev version criteria (#2636)

• update all CI actions (#2460)

packaging:

• Add ibllib to test requirements (#3090)

• Relax numpy (#3056)

• Set DEV=True until release (#3036)

• Remove neo top level imports (#2991)

• Improve import times with full (#2983)

• Remove upper bound in scipy dependency for full install (#2956)

• Improve profile imports test (#2955)

• Fix marker collection to work on windows (#2947)

• Remove numba type signature to prevent compilation of numba functions at import (#2932)

• Switch calculate_pc_metrics to compute_pc_metrics for api consistency (#2925)

• Regularised the API from parent_recording to recording in zero_channel_pad (#2923)

• Update curation module to use sorting argument rather than parent_sorting (#2922)

• Fix the new way of handling cmap in matpltolib. This fix the matplotib 3.9 problem related to this. (#2891)

• Protect hdbscan import in TdCClustering (#2876)

• Pin matplotlib dependency (#2866)

• Unpin sphinx and add networkx dependency. (#2861)

• Add safeguard for probeinterface use of read_spikegadgets (#2833)

• Update codecov badge (#2481)

testing:

• Add a couple of seeds in tests (#3189)

• Add test_extractors to full test with codecov (#3141)

• Add streaming extractor tests to main test script (#3129)

• Add plexon2 tests action for install wine and drop full (linux only) tests (#3101)

• Mark failing sorter test on Windows*Python3.12 as xfail (#3091)

• Add macos and windows to cache cron jobs (#3075)

• Remove duplicate function from common test suite. (#3058)

• Numpy 2.0 cap Fix most egregorious deprecated behavior and cap version (#3032)

• Remove unittest from postprocessing (#2984)

• Fix deepinterpolation tests (#2971)

• Make easier env vars for local testing (#2951)

• Fix marker collection to work on windows (#2947)

• Remove unused imports in ibl test (#2939)

• Add tests for windows and mac (#2937)

• Add Mac skip for plexon2 Neo test (#2931)

• Remove mearec from testing functions (#2930)

• Remove cache folder (#2927)

• Remove toy_example from test codebase (2) (#2920)

• Remove toy example from test codebase (1) (#2916)

• Fix codecov testing (#2777)

• Speed up test_sorting_s3_nwb_zarr (#2767)

• Fix full tests (#2665)

• Fix typo in fast templates test (#2651)

• update all CI actions (#2460)

Contributors:

• @AnujVader

• @Ashkees

• @Djoels

• @DradeAW

• @HDClark94

• @JoeZiminski

• @NinelK

• @RobertoDF

• @alejoe91

• @bagibence

• @chrishalcrow

• @cwindolf

• @gkBCCN

• @h-mayorquin

• @jakeswann1

• @jsiegle

• @khl02007

• @manimoh

• @naterenegar

• @nikhilchandra

• @paulrignanese

• @remi-pr

• @samuelgarcia

• @yger

• @zm711