Segments is a web application designed both as an artistic tool and an environmental audio classifier.
It enables users to record, analyze, visualize and more crucially, resynthesize surrounding soundscapes.
It is based on a combination of AI-powered analysis with algorithmic synthesis.
Segments transform, reorganize and finally render sonic environments into new auditory structures. It is a scientific instrument and a sound design tool, where attentive listening merges with auralization creativity.
Through a growing list of playback modes (algorithms), users can reframe field recordings (or any other audio material) into narrative arcs, stochastic reorganizations and spectral transformations.
Liminal Territories
Segments is based equally in the conceptual use of algorithms and the phenomenology of sound, followed by reorganization and resynthesis in both of these domains.
Through reorganization hidden structures emerges, patterns rises on the surface and unnoticed details are brought to the foreground.
Sometimes, what was once background becomes an aesthetic focus.
The composer creates a dialogue between nature’s proto-musical sound material and audio signal processing, discovering new musical potentials already present in the soundscape.
Key Features
Audio Analysis: Classify audio into 521 everyday labels using MediaPipe's YAMNet model, bucketed into biophonic or anthropophonic categories.
Rendering Modes: Create composite renders with modes like, Lorenz attractor, Markov chain, Random mix or Emotion-based and many more.
Multiple Renders: Render and export up to 8 channels based on custom user rules.
Real-Time Processing: Analyze live microphone input or uploaded audio files (WAV/MP3).
Custom Model Training: Train new sound labels in-browser using TensorFlow, saved to localStorage. This is separate from the YAMNet model and does not modify it.
MIDI Integration: Map sound labels to MIDI notes, for controlling synthesizers or IoT devices.
Conceptual Reverb: Add Reverb to the full render file, by category (Biophony / Anthropophony) or by label.
Visualization: View analysis results as tables and pie charts for intuitive insights.
NEW: Musicality Analyzer: A custom three-factor model for analyzing the musicality of renders. Includes a detailed sound analysis.
NEW: Spectral Synthesis: A tool for further spectral editing, using spectral smear, harmonic boost and more.
NEW: Overtones Generator: A tool to introduce up to 12 new harmonics to the render file.
MediaPipe YAMNet vs. TensorFlow Custom Models
Segments uses two distinct AI models for audio analysis: the pre-trained MediaPipe YAMNet model and a TensorFlow-based custom model for user-defined labels. Below is a comparison of their roles and capabilities:
Feature
MediaPipe YAMNet
TensorFlow Custom Model
Purpose
Pre-trained model for classifying audio into 521 fixed everyday sound labels (e.g., "Bird," "Car Horn"), bucketed into Biophony or Anthropophony categories.
User-trained model for recognizing new, custom sound labels (e.g., "Coffee Machine") not included in YAMNet's label set.
Training
Fixed and pre-trained; cannot be modified or extended with new labels by users.
Trainable in-browser using 5–20 WAV clips (16 kHz, mono, 1–5 s). Users define new labels, and the model is trained independently of YAMNet.
Label Set
521 pre-defined labels, non-editable.
User-defined labels, created via the "Train on These Results" button.
Storage
Embedded in the application, not stored locally.
Saved to browser's localStorage as JSON weights; persists across reloads.
Integration
Provides baseline classification for all audio analysis tasks.
Supplements YAMNet by adding custom labels, which can be used in rendering or MIDI mapping.
Reset
Not applicable (fixed model).
Reset using Ctrl+Shift+X to clear custom model data.
Note: The YAMNet model is used for all initial audio classifications, while the TensorFlow custom model allows users to extend functionality with new sound labels. Custom labels do not modify or add to YAMNet's fixed label set but operate as a separate, parallel model.
How to use Segments
Getting Started
Wait for Model Loading: Ensure the status window (top right) shows "Model loaded".
Drag & Drop Files: Upload WAV or MP3 files via the "Audio Folder Analysis (≤ 10 MB each) ", "Single File (any size)" or "RealTime (up to 24h)" sections.
Render Audio: Select which source method to render (Playback Setting -> Source). Folder, Single, Realtime or a combination of sources. All three input methods can be rendered simultaneously.
Render Audio: Select a playback mode (Playback Setting -> Main Mode -> e.g. "Bounce Chaos") and click "Render". A new file(s) is created based on the selected playback mode and other settings (Pitch, Volume, Reverb etc). Re-render with a different algorithm (Main Mode) to notice the differences.
MIDI Mapping: [optionally] Assign labels (from YAMNet or custom model) to MIDI notes in the "MIDI Mapper" section to control external devices.
View Results: Check the "Folder Analysis Results," "Single File Recording Analysis," or "Real-Time Analysis" sections for tables and pie charts showing top labels, categories and confidence scores.
Optional Custom Training: In "Train Custom Sounds" section, train your personal model with new labels. This creates a separate custom model, not an update to YAMNet.
Training Custom Models (TensorFlow)
Segments allows in-browser training of custom sound labels using TensorFlow, independent of the YAMNet model.
Analyze Clips: Upload clips via folder, single file, or real-time analysis (processed by YAMNet).
Train: Click the green "Train on These Results" button under the relevant analysis table, enter a label (e.g., "Coffee Machine"), and train in-browser with TensorFlow.
Add Negative Examples: Optionally upload negative samples to improve model accuracy.
Save Model: If accuracy ≥ 70%, the model auto-saves to localStorage. Use "Save Model" to export as JSON or "Load Model" to import.
Reset Model: Press Ctrl+Shift+X to reset the custom model (does not affect YAMNet).
Custom Model Architecture
Structure: 1-D CNN → Global MaxPool → Dense(16) → Dense(2-softmax)
Augmentation: ±10% time-stretch, ±5 dB pink-noise, ±3 dB gain
Validation: 20% hold-out, target ≥ 70% accuracy
Storage: JSON weights in localStorage
Note: Custom training creates a new model separate from YAMNet. Use headphones to avoid feedback during training or recording. Ensure clips are clean and representative of the target sound.
Rendering Audio
Segments allows to create composite audio mixes from analyzed segments using various playback modes, leveraging labels from YAMNet or the custom TensorFlow model. Each mode is a sorting / sequencing strategy.
Modes with [L] are available for looping (Read about Looping further).
Realtime: Render only recordings from Realtime Input.
Folder and Single: Combo 1.
Folder and Realtime: Combo 2.
Single and Realtime: Combo 3.
Playback Modes (the list is growing...)
Playback Mode Parameters
Many of the following modes have additional parameters to control the algorithm: Playback Settings > Advanced Settings
🎛️ Playback Modes – Complete Guide
Each playback mode is a unique algorithm that reorders and reorganizes your audio segments.
Modes marked with [L] support looping –
they can generate new variations when "Evolving Renders" is enabled.
📊 Sorting & Basic
low→high
What it does: Arranges segments from lowest to highest frequency.
All Man‑made sounds come first (sorted low→high), followed by all Natural sounds (also sorted low→high).
Parameters: None. Uses pre‑computed frequency values.
high→low
What it does: Reverse of low→high. Arranges from highest to lowest frequency.
All Man‑made sounds come first (sorted high→low), followed by all Natural sounds (also sorted high→low).
Parameters: None. Uses pre‑computed frequency values.
common→rare
What it does: Orders segments from the most common label to the rarest label.
Labels that appear frequently in your analysis are placed first, then progressively rarer labels.
Parameters: None. Uses label frequency from analyzed files.
rare→common
What it does: Reverse of common→rare. Orders segments from the rarest label to the most common.
Rare sounds are featured first, building up to the most frequent sounds.
Parameters: None. Uses label frequency from analyzed files.
random [L]
What it does: Completely shuffles all segments randomly.
The Randomness slider controls how much shuffling occurs:
0% → Sorted by frequency (low→high, Man‑made first)
100% → Fully random order
Between 0–100% → Blends sorted and random order
Parameters: Randomness slider (0–100%).
alt [L]
What it does:Alternates between Natural and Man‑made segments.
Creates an A‑B‑A‑B pattern that switches between biophonic and anthropophonic sounds.
Parameters: Randomness slider (affects how strictly the alternation is maintained).
user
What it does: Runs a custom JavaScript function you write.
This gives you full control over how segments are ordered.
Parameters: Write a function customOrder(files) in the textarea.
The function must return an ordered array of segments.
🔍 Filtering
label
What it does: Filters segments by label keyword.
Only segments whose label matches your keyword(s) are included.
Parameters: Enter keywords in "Label Filter" field.
Use commas for OR logic: bird, rain, wind includes any segment with any of these labels.
exclude-label
What it does: The opposite of label. Excludes segments whose label matches your keyword(s).
Everything else is included.
Parameters: Enter keywords in "Label Filter" field.
Use commas for OR logic: traffic, siren removes any segment with these labels.
sparse
What it does: Selects a random subset of segments based on a density percentage.
Creates a sparse, "thinned out" render where only some segments play.
Parameters:"Density" slider (5–100%).
100% = all segments, 50% = half the segments randomly selected.
frequency-range
What it does: Filters segments by frequency range.
Only segments whose estimated frequency falls within the specified range are included.
Parameters:"Min (Hz)" and "Max (Hz)" fields.
Example: 100 Hz – 5000 Hz keeps only mid‑range sounds.
emotion-valence
What it does: Filters segments by emotional tone.
Uses a pre‑configured list of keywords to classify sounds as Positive, Neutral, or Negative.
Parameters:"Emotional Tone" dropdown (Positive / Neutral / Negative).
If no match is found, Natural sounds default to Positive, Man‑made to Neutral, and alarms/sirens to Negative.
🌀 Chaos & Attractors
lorenz [L]
What it does: Uses the Lorenz chaotic attractor to generate an unpredictable, organic sequence.
The famous "butterfly effect" chaos creates smooth, flowing transitions with surprising jumps.
How it sounds: Smooth, flowing, with occasional unexpected leaps. Natural‑sounding yet unpredictable.
What it does: Uses Chua's circuit – a double‑scroll attractor that creates a "figure‑8" pattern.
Excellent for A‑B‑A structures and dual‑state switching between two sonic worlds.
How it sounds: Alternates between two contrasting "scrolls" (e.g., Natural vs Man‑made) with dramatic transitions.
What it does: Uses the Ikeda map – a laser cavity chaos model that creates crescent patterns.
Generates a shimmering, rotating, "spinning" texture.
How it sounds: Rotational, shimmering textures with layers that spiral inward and outward.
Inner layer = Natural sounds, Outer layer = Man‑made sounds.
Parameters:u (0.9 default). Controls the chaos intensity.
henon-map [L]
What it does: Uses the Henon map – a 2D discrete chaos system that creates a "horsehead" pattern.
Ideal for binary/call‑response patterns and symmetrical structures.
How it sounds: Call‑response patterns, symmetrical structures, with a natural "breathing" quality.
Parameters:a (1.4 default), b (0.3 default). Control the attractor shape.
rossler-attractor [L]
What it does: Uses the Rossler attractor – a folded‑band chaos system.
Creates cyclic, hypnotic patterns with sharp "spikes" that create rhythmic emphasis.
How it sounds: Hypnotic, cyclic, with sudden "beats" or spikes. Good for trance‑like textures.
Parameters:a (0.2 default), b (0.2 default), c (5.7 default).
chaos-bounce [L]
What it does: Uses the Lorenz attractor but scales the jumps using the Randomness slider.
At 0% it's nearly sequential; at 100% it's chaotic leaps.
How it sounds: Can be gentle and flowing (low randomness) or wild and unpredictable (high randomness).
Parameters:"Chaos Bounce Randomness" slider (0–100%).
Also uses Lorenz parameters from the Chaos section.
⬡ Rhythmic & Cellular
euclidean [L]
What it does: Arranges segments using a Euclidean rhythm pattern.
Segments are placed at onset times determined by the Euclidean algorithm – the same logic behind many traditional rhythms.
How it sounds: Rhythmic, structured, with a "groove" that depends on the Steps and Pulses settings.
What it does: Uses a 1‑D cellular automaton (like Conway's Game of Life, but 1D).
Each bar, a segment is either kept, muted, or replaced by its spectral neighbour based on the chosen rule.
How it sounds: Evolving, generative, with patterns that grow, shrink, and mutate over time.
What it does: Sorts segments by duration and distributes them within a cycle (default 10 seconds).
Creates a "tidal" ebb‑and‑flow effect where segments wash in and out.
How it sounds: Flowing, wave‑like, with segments appearing in pulses.
Parameters:Cycle Duration (in seconds, default 10).
Randomness slider adds variability to start times.
🎵 Spectral & Formant
spectral-flow
What it does: Sorts segments by spectral brightness (RMS of the upper‑half spectrum).
Creates a flow from "dull" to "bright" or vice‑versa.
How it sounds: A gradual shift from dark, muffled sounds to bright, crisp sounds.
Parameters: None directly. Uses FFT analysis on each segment.
spectral-stutter [L]
What it does: Combines spectral morphing with rhythmic stuttering.
First, it blends adjacent segments spectrally (morph), then repeats each segment at the chosen BPM.
How it sounds: Stuttering, shimmering textures with spectral glides between sounds.
Parameters:BPM (30–300). Randomness slider for shuffling.
rhythmic-stutter [L]
What it does: Repeats each segment to fill a full beat at the chosen BPM.
Creates a "looper" effect where each sound plays for the duration of a beat.
How it sounds: Rhythmic, pulsed, with each sound repeating to fill the beat.
Parameters:BPM (30–300).
formant-ladder
What it does: Sorts segments by average frequency, then picks an evenly‑spaced "ladder" (2–12 steps).
Creates an ascending or descending vocal formant‑like effect.
How it sounds: Like vocal formants moving up or down – "ee‑ay‑ah‑oh‑oo" style transitions.
Parameters:Formant Steps (2–12). More steps = smoother ladder.
formant-reorder [L]
What it does:Pitch‑shifts every segment by a chosen ratio, sorts by the new formant frequencies, then optionally shuffles.
Creates a dramatic "formant ladder" with pitch‑shifted sounds.
How it sounds: A pitched‑up or pitched‑down version of the formant ladder – more extreme than regular formant‑ladder.
Parameters:Formant Shift (pitch ratio). Randomness slider for shuffling.
📖 Compositional & Narrative
narrative-journey [L]
What it does: Creates a dramatic narrative arc:
Calm (Natural, low frequency) → Tension (Man‑made, ascending) →
Climax (hybrid, complex labels) → Resolution (Natural, high frequency, descending).
How it sounds: A story in sound – peaceful beginning, building tension, chaotic climax, peaceful resolution.
Parameters:Narrative Balance (affects the mix of Natural vs Man‑made). Randomness slider for shuffling within sections.
ecosystem-simulation [L]
What it does: Simulates an ecosystem where sounds "compete" based on category rarity.
"Predators" (rare categories) "consume" order positions from "prey" (common ones).
How it sounds: Organic, evolving, with rare sounds pushing their way through common ones.
Parameters:Ecosystem Mutation (0–100%). Higher values = more frequent "mutations" where segments swap.
cultural-geography [L]
What it does: Takes you on a geographical journey through soundscapes:
Rural → Suburban → Urban → Industrial → Natural.
How it sounds: A sonic tour from quiet countryside through busy city to peaceful nature.
Parameters:Cultural Direction (Rural→Natural, Natural→Rural, Bidirectional).
Max per Zone (how many segments from each zone).
maslow-hierarchy [L]
What it does: Orders segments by Maslow's hierarchy of needs:
Physiological → Safety → Belonging → Esteem → Self‑Actualization.
How it sounds: A philosophical journey from basic sounds (breathing, heartbeats) to transcendent sounds (meditation, harmony).
Parameters:Maslow Direction (Ascending / Descending).
Segments per Level (how many from each level).
ontology-traversal [L]
What it does: Organizes sounds in a hierarchical ontology (Natural → Weather → Rain → etc.)
and traverses it using your chosen method.
How it sounds: A structured, logical progression through sound categories, like exploring a tree of knowledge.
Parameters:Ontological Method (Depth‑First / Breadth‑First / Preorder / Postorder / Inorder).
Max Depth, Max per Leaf.
🔬 Aesthetic Friction
exposition [L]
What it does:Exposes the AI's listening biases by foregrounding uncertainty and contradictions in classification.
Segments with low confidence or competing labels are emphasized, making the system's hidden assumptions audible.
How it sounds: You hear the AI's uncertainty – sounds it's not sure about, or sounds that confuse it.
The "coded ear" becomes audible.
Parameters: Exposure Depth – how much of the uncertainty is emphasized (0–100%).
Show Uncertainty – include low‑confidence sounds.
Temporal Violence – make the 0.96‑second cuts audible as rhythmic elements.
Expose Contradictions – foreground sounds with competing label classifications.
subversion [L]
What it does:Subverts the AI's intended purpose by amplifying errors, juxtaposing unrelated categories, and creating dialectical tension between human and machine perception.
How it sounds: Confrontational, challenging, unexpected. Sounds that should be separate are forced together; the AI's mistakes become musical material.
Parameters: Subversion Intensity – how radical the subversion is (0–100%).
Error Refinement – how much to amplify low‑confidence sounds.
Dialectical Tension – create friction between "human‑like" and "machine‑like" perception.
Categorical Juxtaposition – force unrelated categories to play side‑by‑side.
🧱 Material
material-blend
What it does:Interpolates between material categories.
Uses a 2D space: X‑axis (Organic ↔ Synthetic) and Y‑axis (Soft ↔ Hard).
Selects segments that match your chosen material "region."
How it sounds: A smooth transition through material textures – from wood to metal to glass to fabric, etc.
Parameters: X (Organic→Synthetic) – move from natural to artificial materials.
Y (Soft→Hard) – move from soft/pliable to hard/rigid materials.
Blur – how wide the material selection is (sharp focus vs broad blend).
Density – how many segments to select.
Preset Buttons – Wood, Metal, Glass, Fabric, Stone, Liquid, Plastic, Paper.
⚙️ Other
markov [L]
What it does: Uses a Markov chain – a probabilistic model where the next segment depends on the current one.
Transitions between categories determine what comes next.
How it sounds: Flowing, connected, with smooth transitions that feel logical yet unpredictable.
What it does: Simulates quantum behavior – like a wave function collapsing upon observation.
Higher‑intensity segments have a higher probability of being selected, but randomness ensures variety.
How it sounds: Probabilistic, "superposition" of possibilities. A blend of structure (favoring loud/intense sounds) and randomness.
Parameters: None directly. Uses RMS intensity as the "wave function" probability.
spectral-transfer
What it does:Transfers spectral characteristics from a source label to a target label.
The "color" or "tone" of the source sound is applied to the target sounds.
How it sounds: Target sounds take on the spectral "flavour" of the source – e.g., traffic sounds made to sound like wind, or vice‑versa.
Parameters: Source Label – the label to take spectral characteristics from.
Target Label – the label to apply the spectral characteristics to.
Transfer Amount – how much of the source spectrum to apply (0–100%).
Preserve Transients – keep the attack clarity of the target.
Match Duration – adjust target duration to match the source.
User Custom Algorithms Examples
Alphabetical Order (A-Z): function customOrder(files) {
return files.slice().sort((a, b) => {
const nameA = a.name || '';
const nameB = b.name || '';
return nameA.localeCompare(nameB);
});
}
Random Shuffle: function customOrder(files) {
const shuffled = files.slice();
for (let i = shuffled.length - 1; i > 0; i--) {
const j = Math.floor(Math.random() * (i + 1));
[shuffled[i], shuffled[j]] = [shuffled[j], shuffled[i]];
}
return shuffled;
}
Key Considerations: Function Signature and Return Value must be named customOrder: The function must be defined as function customOrder(files) { ... } because generateBuffer expects to call customOrder explicitly.
Input: The files parameter is an array of segment objects, each with properties like: audioBuffer: An AudioBuffer (Web Audio API) with duration, sampleRate (16 kHz), numberOfChannels, and getChannelData(ch). category: String, either "Natural" or "Man-made". name: String, the file name (e.g., "bird.wav"). topLabel: String, a primary label (e.g., "bird_call"). topCategories: Array of objects with categoryName (e.g., [{ categoryName: "bird" }, ...]). frequency: Number, assumed for frequency-based sorting (e.g., in 'low→high').
Output: Must return an array of segment objects (a subset or reordering of files). The array must contain valid segments (each with an audioBuffer).
Looping Controls - Evolving Playback
Evolving Renders: If checked, creates a new random render with each new loop (loop time is specified in Render Length) based on the selected mode. Supported in selected playback modes.
Random Mode: Requires Evolving Renders checked. If checked, for each new loop a random playback mode is selected, along with a new random render. Supported in selected playback modes.
Check Both: For endless, non-repeatable playback.
Label Playback Mode
Label Select: Enter the label name to create a render only based on the choosen label.
Label Exclude: Enter label name to exclude it from render.
Rendering Options (Main Controls Section)
Volume: Adjust playback volume.
Time-Stretch: Adjust playback speed (e.g., 1x for normal, 0.5x for slower).
Example: label bird and freq > 1000 or label insect and duration < 2 → 1,2 (all conditions must be true to return segments)
Example: category natural and freq 500-2000 or rms > -25 → 3,4,5
Browsers do not directly support multi-channel audio to drive the multiple renders into multiple speakers in real-time.
Howevever, the multiple renders files can be uploaded to your DAW and driven to selected outputs if your audio interface support it.
There is a known bug which I am working to fix it. It does not affect the core functionality of the app, however it can be annoying.
Importing large files (mostly in Single file but sometimes in Folder Input files too), crushes the audio engine (only the first render).
When this happens, pressing Play may not start playback immediately.
FIX: Press Stop Button > Render again > Press Play. Maybe you need to repeate these steps for 2 or 3 times before audio is ready to play. All following renders, using the same files and while trying other playback modes, function normally.
Using MIDI Functions
Segments maps sound labels (from YAMNet or custom TensorFlow model) to MIDI notes, to control synthesizers, DAWs, or IoT devices.
Steps to Set Up MIDI
Select MIDI Port: Choose an output port in the "MIDI Mapper" section.
Add Mapping: In the "Live MIDI Map" or "Pre-Map Sounds to MIDI" section, assign a label (from YAMNet or custom model) to a MIDI Note, CC, or Channel.
Common Sounds: Pre-map frequently detected YAMNet labels (e.g., "Bird," "Traffic").
Custom Sounds: Add custom-trained labels and assign MIDI parameters.
Save Mappings: Click "Save All Mappings" to store configurations.
Re-scan MIDI: Refresh available MIDI ports if devices change.
Note: Ensure your MIDI device or software is connected and recognized by the browser before mapping.
Practical Scenarios and Applications
Nature Conservation
Use Case: Use Segments to monitor a natural sonic environment. Record 10 minutes of soundscape, analyze it and find the percentage of Biophony sounds (birds, wind) vs.Anthropophony (distant traffic) in a specific area. Pie charts visualize ecosystem health. Hint: Train a custom TensorFlow model to detect a rare frog species, aiding conservation efforts.
Music and Sound Design
Use Case: Upload urban sound recordings, analyze and render a Markov chain-based mix of traffic and rain sounds. Then map a custom "Car Horn" label (trained via TensorFlow) to MIDI Note C3 to trigger a synth in their DAW, creating a dynamic soundscape for a film score.
Music Creation: Experimental Live Performance
Use Case: Use Segments's real-time analysis during a live set. Stream microphone input and the model detects sounds like "Applause" and "Footsteps" or your custom training labels. These are mapped to MIDI CCs to modulate synth parameters in e.g Ableton Live.
Smart Home and IoT
Use Case: Train a custom "Doorbell" label using TensorFlow and maps it to a MIDI CC to trigger smart lights. Segments detects the doorbell in real-time, automating the lighting response.
Film and Games
Use Case: Upload location recordings, filters for YAMNet's "Footsteps" label using Render by Label and downloads a composite WAV for a game's Foley library.
Spatial Audio
Use Case: Use the multi render section to generate up to 8 different audio channels. Upload to your DAW and use a multi-output audio interface to drive selected speakers.
File Formats and Specifications
Use Case
Format
Bits / Channels
Sample Rate
Max Size
Length Notes
Folder / Single Analysis
WAV, MP3
16–24 bit, mono/stereo
Auto-converted to 16 kHz mono
10 MB/file
≥ 0.1 s
Real-Time Stream
Web-Audio PCM
Float32, mono
16 kHz
—
Continuous
Custom-Training Samples
WAV (preferred)
16 bit, mono
16 kHz
10 MB/file
1–5 s ideal
Pie Charts
Purpose:Visualizes the distribution of sounds.
Sound Event Distribution Pie: Analyze the sonic environment for individual sounds (human or natural, with labels).
Background Pie: Visualizes the distribution of dominant, steady background sounds in an audio file or real-time input, focusing on persistent soundscapes rather than transient events.
Background Window (1-30s): Sets the time window for detecting dominant, steady background sounds, filtering out short, one-shot events like bird chirps or claps.
Min Hits: Specifies the minimum number of times a sound label must appear within this window to be considered a significant background element.
Real-Time Constraints: Long recordings (e.g., 1440 minutes) may strain browser memory.
The Three-Factor Model of Emergent Musicality
Segments incorporates a sophisticated three-factor model for analyzing how musicality emerges through hybrid human-algorithmic listening.
Key Insight: Musicality (M) is not discovered as an inherent property of soundscapes. Instead, it emerges through interaction between sonic material, computational transformation, and compositional practice.
The Musicality Formula
M = ƒ(ΔOD, ΔED, ΔCA)
Where M > 0 requires ΔOD > threshold₁, and M is bounded by ΔCA
The Three Factors Explained
Factor
What it is
What it measures
ΔOD (Organizational Density)
The degree of internal coherence and statistical interdependence among sonic events within a soundscape, independent of musical intention or convention.
The capacity of a soundscape to generate new structural relationships when subjected to computational analysis, revealing patterns not easily accessible to direct listening.
Latent category classification
Semantic clustering potential
Temporal segmentation capacity
Spectral resynthesis potential
ΔCA (Compositional Affordance)
The degree to which structures emerging from ΔOD and ΔED become actionable as compositional material in hybrid human-algorithm workflows.
Decision Clarity: Speed and confidence of compositional decisions
Vocabulary Development: Transferable strategies and labels
Cognitive Load Management: System enables rather than overwhelms
System Legibility: Predictable and interpretable behavior
Aesthetic Productivity: Engagement produces valued material
How Musicality Emerges
Organizational Density (ΔOD): Forms the foundational layer of measurable sonic structure
Emergent Differentiation (ΔED): Enabled and constrained by ΔOD, reveals computational transformation potential
Compositional Affordance (ΔCA): Enabled and constrained by ΔED, measures practical accessibility for composition
Musicality (M) emerges only when all three factors are present and sufficiently high.
How the App Measures These Factors
The Enhanced Audio Analyzer section uses various audio analysis techniques to compute these factors:
ΔOD Measurement: Uses periodicity analysis (Zero Crossing Rate) and spectral concentration (Spectral Centroid) to quantify organizational density
ΔED Measurement: Analyzes frequency spread and dynamic range to estimate emergent differentiation potential through computational processing
ΔCA Measurement: Evaluates signal clarity and spectral balance to determine compositional affordance and practical usability
Overall Musicality (M): Calculated as: M = (ΔOD × 0.4) + (ΔED × 0.35) + (ΔCA × 0.25), with ΔCA as the bounding constraint
Note about ΔCA measurement: To measure true ΔCA, the app would need to evolve into an interactive platform that logs composer actions, corrections, iterations and vocabulary use over multiple sessions, thereby measuring the collaboration itself. However, it uses the recording's features to infer the probable potential for emergent differentiation and compositional collaboration.
Technical Implementation
Behind the scenes, the analyzer computes these key audio features:
Metric
Purpose
Typical Values
Spectral Centroid
The "center of mass" of frequencies, indicating brightness (influences ΔOD)
Frequency distribution width around centroid (influences ΔED)
Low: Pure tones | High: Noise, complex textures
Spectral Flatness
Tonal vs. noise character (influences ΔED)
0.0-0.3: Tonal | 0.3-0.7: Mixed | 0.7-1.0: Noisy
Zero Crossing Rate (ZCR)
Indicates periodicity and noisiness (influences ΔOD)
High: Noisy | Low: Tonal/periodic
Crest Factor
Peak-to-average ratio (influences ΔCA)
1.0-2.0: Compressed | 2.0-5.0: Normal | >5.0: Percussive
Limitations and Known Issues
File Size: Maximum 10 MB per file in Folder Analysis to ensure browser performance and proper analysis. However, and if your system can handle it, any file size is valid for analysis and creative applications, but without expecting proper label results.
Train Segments: User need to create and save a custom model in the Training Section first, then train individual segments from analysis.
Sample Rate: All audio is converted to 16 kHz mono for analysis, which may affect high-frequency content. Final render is back at 16Bit/44.1Khz.
Training Accuracy: Custom TensorFlow models require ≥ 70% accuracy; noisy or insufficient samples may reduce performance.
MIDI Compatibility: Requires browser support for Web MIDI API and connected MIDI devices.
Memory Constraints: Long recordings may strain browser memory. Be patient when uploading large files or folders. Use 'Clear' > "Render' > 'Clear' > "Render' if playback fails to start.
Audio Engine: Importing large files sometimes crushes the audio engine. Press in this order: Stop -> Render -> Play up to 3 times to restore playback.
Future
Multi-output support to spatialize segments.
Extended midi functions.
OSC support.
Online user base with custom Playback algorithms.
Standalone application (Desktop/iOS/Android).
Developed:
To support and extend the creative process.
To support my PhD thesis 'Algorithmic Soundscape Compostion' at HMU.
For ihearcolors.online
For tokeno.net
By Dimitris Barnias 2025
dbarnias@gmail.com | Contact here for questions, ideas, comments |