René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[Bug]: Long audio files result in incomplete spectrogram visualizations","articleBody":"### Bug summary\n\nWhen using matplotlib to generate spectrogram visualizations of audio files, if the audio file is too long, the spectrogram portion of the plot becomes blank towards the latter half, while the waveform continues to be displayed properly.\n\n### Code for reproduction\n\n```python\nimport torchaudio\r\nimport torchaudio.transforms as transforms\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\nimport scipy.interpolate\r\nimport librosa.display\r\nimport matplotlib.gridspec as gridspec\r\n\r\nfrom tqdm import tqdm\r\nfrom concurrent.futures import ProcessPoolExecutor, as_completed\r\n\r\n\r\ndef compute_spectrogram_per_channel(channel_waveform, sample_rate):\r\n    # Create transformer to convert waveform to spectrogram\r\n    spectrogram_transform = transforms.Spectrogram(n_fft=2048, hop_length=256)\r\n\r\n    # Apply the transformer\r\n    spectrogram = spectrogram_transform(channel_waveform.unsqueeze(0))\r\n\r\n    amplitude_spectrogram = np.sqrt(spectrogram)\r\n    db_spectrogram = librosa.amplitude_to_db(amplitude_spectrogram[0].numpy(), ref=np.max)\r\n    db_spectrogram = np.clip(db_spectrogram, a_min=None, a_max=0)  # clip to 0dB\r\n\r\n    # Set new log scale\r\n    num_freqs, num_frames = db_spectrogram.shape\r\n    min_freq = 1  # human hearing range in Hz\r\n    max_freq = sample_rate / 2  # Nyquist frequency\r\n    frequencies = np.linspace(min_freq, max_freq, num=num_freqs)\r\n\r\n    # Create a new scale\r\n    log_scale = np.log10(frequencies)\r\n    linear_scale = np.linspace(np.log10(min_freq), np.log10(max_freq), num=num_freqs)\r\n    scale_ratio = 0.75  # adjust this parameter to control the ratio of log scale and linear scale\r\n    new_scale = scale_ratio * log_scale + (1 - scale_ratio) * linear_scale\r\n\r\n    new_db_spectrogram = np.empty_like(db_spectrogram)\r\n\r\n    # Apply interpolation for each frame\r\n    for frame in tqdm(range(num_frames)):\r\n        interpolator = scipy.interpolate.interp1d(log_scale, db_spectrogram[:, frame])\r\n        new_db_spectrogram[:, frame] = interpolator(new_scale)\r\n\r\n    return channel_waveform.t().numpy(), db_spectrogram, new_db_spectrogram\r\n\r\n\r\ndef plot_spectrogram(waveforms, new_db_spectrograms, audio_duration):\r\n    num_channels = len(waveforms)\r\n\r\n    # Create a plot, set the background to black and adjust the size based on audio duration\r\n    plt.figure(figsize=(max(audio_duration * 2, 10), 8), facecolor=\"black\")\r\n\r\n    # Dynamically create subplots based on the number of channels\r\n    gs = gridspec.GridSpec(2 * num_channels, 1, height_ratios=[1] * num_channels + [5] * num_channels)\r\n\r\n    # Loop through each channel to plot the waveform and spectrogram\r\n    for i in range(num_channels):\r\n        # Plot the waveform\r\n        ax_waveform = plt.subplot(gs[i])\r\n        ax_waveform.plot(waveforms[i], color=\"#4BF2A7\")\r\n        nonzero_indices = np.where(waveforms[i] != 0)[0]  # Find indices of non-zero values\r\n        ax_waveform.set_xlim(nonzero_indices[0], nonzero_indices[-1])  # Set x limit to range of non-zero values\r\n        ax_waveform.axis(\"off\")\r\n\r\n        # Plot the spectrogram\r\n        ax_spectrogram = plt.subplot(gs[i + num_channels])\r\n        ax_spectrogram.imshow(new_db_spectrograms[i], origin=\"lower\", aspect=\"auto\")\r\n        ax_spectrogram.axis(\"off\")\r\n\r\n    plt.subplots_adjust(left=0, right=1, top=1, bottom=0, wspace=0, hspace=0)  # Adjust to remove borders and gaps\r\n    plt.savefig(\"spectrogram.jpeg\", facecolor=\"black\", bbox_inches=\"tight\", pad_inches=0)  # Save the figure\r\n\r\n\r\ndef main(file):\r\n    # Load the audio file\r\n    waveform, sample_rate = torchaudio.load(file)\r\n\r\n    num_channels = waveform.shape[0]\r\n    audio_duration = waveform.shape[1] / sample_rate  # Calculate audio duration\r\n\r\n    with ProcessPoolExecutor() as executor:\r\n        futures = {\r\n            executor.submit(compute_spectrogram_per_channel, waveform[ch], sample_rate): ch\r\n            for ch in range(num_channels)\r\n        }\r\n        waveforms = [None] * num_channels\r\n        db_spectrograms = [None] * num_channels\r\n        new_db_spectrograms = [None] * num_channels\r\n        for future in as_completed(futures):\r\n            ch = futures[future]\r\n            waveforms[ch], db_spectrograms[ch], new_db_spectrograms[ch] = future.result()\r\n\r\n    # Adjust figure width based on audio duration\r\n    plot_spectrogram(waveforms, new_db_spectrograms, audio_duration)\r\n\r\n\r\n# Run the main function\r\nif __name__ == \"__main__\":\r\n    file = \"long.wav\"\r\n    main(file)\n```\n\n\n### Actual outcome\n\nWhen visualizing an audio file of significant length, the spectrogram does not render correctly for the entire duration of the audio. The latter part of the spectrogram is blank and contains no information, which doesn't match with the waveform visualization, which continues to display normally.\r\n\r\nlong wave\r\n![spectrogram](https://github.com/matplotlib/matplotlib/assets/59153990/1abda313-8ca3-4f8d-b398-3bb1b5290b99)\r\n\r\nshort wave\r\n![spectrogram](https://github.com/matplotlib/matplotlib/assets/59153990/23635ce8-6156-4d5e-8cd1-8f7dbc0fedcc)\r\n\n\n### Expected outcome\n\nThe spectrogram should be consistently rendered for the entire duration of the audio file, matching the waveform visualization. Regardless of the length of the audio file, the spectrogram should be complete and not become blank at any point.\n\n### Additional information\n\n - Other libraries involved: torchaudio, librosa, numpy, scipy\r\n\r\nThis issue occurs regardless of the audio file format or the specifics of the audio content. It seems directly related to the length of the audio file.\n\n### Operating system\n\nWindows \u0026 Ubuntu\n\n### Matplotlib Version\n\n3.7.2\n\n### Matplotlib Backend\n\nTkAgg\n\n### Python version\n\nPython 3.10.10\n\n### Jupyter version\n\n_No response_\n\n### Installation\n\npip","author":{"url":"https://github.com/djkcyl","@type":"Person","name":"djkcyl"},"datePublished":"2023-07-21T04:06:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":13},"url":"https://github.com/26368/matplotlib/issues/26368"}