The program which starts your arecord session is different from the program which is at the receiving end of the pipe. The way I do it in PHP is like this:

and in the recording program I do:
Although php is not python, I assume the principle is not very much different. Note that in my opinion PHP is not suitable processing all the recorded bytes as it is too slow for that.

Note also that your arecord options are not correct. You certainly don't want -v and -f -cd is not correct.

jlinkels

OK thnaks, I'll try to adapt it to python.

I tried the code below:

import subprocess
p1=subprocess.Popen("/usr/bin/arecord -f cd -t wav | /home/ubuntu/dsl/exospython/test4.py", shell=True)


...but I get a permission denied message:

" /bin/sh: /home/ubuntu/dsl/exospython/test4.py: Permission denied
Recording WAVE 'stdin' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo
arecord: begin_wave:1823: write error"

I tried to add "sudo" after before /usr, I also tried with test4.py open or closed but get the same message.

Why???


Thanks for your help.

Usually the fiel doesn't have execute permissions. Do:
I assume you have write permissions in /home/ubuntu otherwise you would not have been able to create test4.py at all.

You might also try to find out how to call the Python program, and specify a script to run. That might be as easy as:
The second message about the write error is a result of the first error.

jlinkels

OK thank-you.

I have succeeded in piping to the python program and actually have done a lot of progress towards implementation of the instructions given by Jiinkels (see below, some of the code is there to help me understand what is happening or in preparation for future development).


The problem is that when I call the function which feeds the circular buffer, circbuffer.append the loop is too slow (the while loop goes on for ages), even calling it with a thread ==>threading.Thread(circbuffer.append(samps_list[count])).start().

I suppose this is due to python being a script language as rightly predicted by JIINKELS. Or is it due to poor coding of the circular buffer?

What can I do? can I modify the code somehow? or if not call a small c program from inside the python program doing the circular buffer ( I don't have the slightest idea how to do that).

Please+++ advise. Thanks


=============================================
import pyaudio
import wave
import sys
import pygame
import numpy
import array
import subprocess
import time
import sys
import os
import threading
pygame.init()

p = os.popen('killall -9 arecord')

chunk = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 2
RATE = 44100
RECORD_SECONDS = 5
WAVE_OUTPUT_FILENAME = "output.wav"


#class RingBuffer builds the ring buffer
class RingBuffer:
def __init__(self,size_max):
self.max = size_max
self.data = []
def append(self,x):
"""append an element at the end of the buffer"""
self.data.append(x)
if len(self.data) == self.max:
self.cur=0
self.__class__ = RingBufferFull
def get(self):
""" return a list of elements from the oldest to the newest"""
return self.data


class RingBufferFull:
def __init__(self,n):
raise "you should use RingBuffer"
def append(self,x):
self.data[self.cur]=x
self.cur=(self.cur+1) % self.max
def get(self):
return self.data[self.cur:]+self.data[:self.cur]

#pop_Ring transforms the hex string in data into an array for future manipulation
def pop_Ring(xdata):
data=xdata
samps = numpy.fromstring(data, dtype=numpy.int16)
return samps

#callRing is a thread to append data to the circular buffer
def callRing(sampsList):
for count in range(len(samps_list)):
#print "hello"
threading.Thread(circbuffer.append(samps_list[count])).start()


all = []
all2 = []
data=[]

#Size of the circular buffer is 44100x3000
circbuffer=RingBuffer(132300)

print "* recording"

#Open arecord and pipe to stdout during 30 seconds.
p1 = subprocess.Popen('/usr/bin/arecord -v -d 30 -r 44100 -t wav -f cd', stdout=subprocess.PIPE, shell=True)

#sampls_list is a list of arrays each one containing a 'chunk' of data
samps_list=[]
n=0
while True:
# Read the nth chunk of bytes
data=p1.stdout.read(chunk)

samps=pop_Ring(data)

#Create List of arrays
samps_list.append(samps)
print len(samps)

#Create ring buffer by taking each array (samps) at a time
threading.Thread(callRing(samps_list)).start()

all.append(data)
# all2.append(samps)
sys.stdout.flush()
n=+1
if data == '':
break
p = pyaudio.PyAudio()
print "* done recording"

#RingBuffer.start()

#stream.close()
p.terminate()

# write data to WAVE file for the moment, recording of cicular buffer under development
data = ''.join(all)
wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
wf.setnchannels(CHANNELS)
wf.setsampwidth(p.get_sample_size(FORMAT))
wf.setframerate(RATE)
wf.writeframes(data)
wf.close()

I can't even read Python... I should do something about my education.

How do you process the samples and append them to the ringbuffer, one by one?

If so, you could try to benefit from how scripts work internally. That is, if you have to perform a copy action 10.000 times and you do it in your own loop that is slow. However, if you manage to use a single instruction to copy the 10.000 things, that might be blazingly fast.

So instead of appending the samples one by one, you should try to do that in chunks of 100kbytes or so. Which means that in a single copy operation you'd do this 100 kbytes. I assume there exists some copyarray function in Python.

The only thing is that your granularity is a bit less as you are storing a few seconds at a time. You can overcome that by increasing the buffer size slightly.

If you are already doing this chunk copy, then, yes, a script might be too slow.

jlinkels

No you are right I was copying one by one.
I will follow your advice and try the other approach.
Thank-you

I'm now concatenating and retrieving large chunks (512 elements) from the ringBuffer at each loop, (where array is the curreng ringBuffer and ringBuf is the new chunk of 512 elements coming from stdin

"...ringBuffer=numpy.hstack((array,ringBuf))
if len(ringBuffer)>MAXSIZEOFRINGBUFFER:
ringBuffer=ringBuffer[512:]..."

and indeed it runs much faster. When I play the ringBuffer after converting it to string and formatting it for .wav the sound is fine if the ringbuffer is not full but once the function which cuts pieces at the start of the buffer (ringBuffer=ringBuffer[512:]...") has started to cut the sound is not good anymore. I wonder if it has something to do with the speed of the loops. I will try to increase the chunk size even more but I would like to understand some points:

1. I understand a .wav file is made of header+[B1B2]_[B3B4] etc where B1B2 is the first 2 bytes of one channel. Now, if I cut a certain number of elements from the array at the beginning (for the circular buffer) am I write to say that the sound will be OK as long as the chunk cut is a multiple of 4 (because I need to cut 2 bytes per channel)?But I don't know if the header is a multiple of 4 does that matter? for example lets say the header is 39 bytes and I cut 44 then the file will start with B6_B7B8 wiil that alter the the sound as the whole sequence reading 2x2 byes will be altered?

2. How fast does arecord 'spit' the data, I suppose it is 44100x4 bytes per second? Should I try to compute the array elements at the same speed? I suppose that if I do it slowly that will be a problem after a while but I do it to rapidly for example by increasing the chunk size what will happen as my while loop breaks if data==0?

3. When I do read(chunk) what happens if the data in the arecord buffer is less than chunk?
for information this is the information I get from arecord when I start it:


"Recording WAVE 'stdin' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo
ALSA <-> PulseAudio PCM I/O Plugin
Its setup is:
stream : CAPTURE
access : RW_INTERLEAVED
format : S16_LE
subformat : STD
channels : 2
rate : 44100
exact rate : 44100 (44100/1)
msbits : 16
buffer_size : 22050
period_size : 5512
period_time : 125000
tick_time : 0
tstamp_mode : NONE
period_step : 1
sleep_min : 0
avail_min : 5512
xfer_align : 5512
start_threshold : 1
stop_threshold : 22050
silence_threshold: 0
silence_size : 0
boundary : 1445068800
"


4. When I want to stop prerecording and start recording it is not very clear to me how to empty the circular buffer into lame and pipe directly from arecord as soon as the circular buffer is empty.


Thanks for you help.

I'm stuck.
I have raw pcm strings in my python program that I would like to pipe (stream) into lame to convert them to mp3 file but don't know how to read the string in python and pipe it into lame.
Can somebody help?
Thanks

I made some progress:

I can now pipe chunks of data into lame (raw format)
and can create an mp3 file successfully by appending them together than piping them to lame (steps 1 and 4 below)

But if I convert each chunk to an array and then back to string the file (steps 2 and 3)
I get has one second of music then one second of static and so on.
Why is that? Could the problem be solved by putting 2 and 3 in a thread rather than in the main program????
Please advise



1. take chunk of raw pcm data from arecord stdout
2. convert to array with numpy.fromstring
3 convert array to string with numpy.tostring
4. pipe string to lame stdin ==>.mp3 file

I put steps 2 and 3 in a thread but I still get the same problem: I get one second of music and once second of saturated static noise.
Jlinkels can you help???
Thanks

No idea, but you should do an incremental approach to solve this problem.

First, save your output to a file instead of piping it into lame. Does that file play? If not, find out what is wrong with your file format.

Does it play? Then cat and pipe this file into Lame. Does that give an error? Then your piping is wrong or you are sending your file into Lame in the wrong way.

Since you says 1 second good, one second bad, my intiuitive reaction would be there is either something wrong with the way you store left/right, or after one second worth of samples you make a mistake in buffer reading or writing which is corrected the next second. You should see a difference when you play with your chunk size.

This is hard to say for someone who doesn't have the code running. Since you have it running it is a matter of trying this, trying that and find a correlation between what you do and what the result is.

Sorry that I cannot be of better help.

jlinkels

This is actually very helpful. I will follow your advice.
Thank-you

I followed your advice and rapidly found the solution:
I was using this instruction to go from string (chunk of raw pcm) to array (numpy array)
1. samps_test = numpy.fromstring(ledata, dtype=numpy.int16)
and then after doing some manipulations I had this instruction to go from array to string:
2. samps_test=samps_test.tostring(),'int16'
and this gave one second of music and one of static noise. Now I found that if I skip the 'int16' then it works fine so instead of 2. =>
3.samps_test=samps_test.tostring()
I don't quite understand why the 'int16' has to be given on instruction 1. but not 2.... (any idea?); anyway this is solved now.

OK now I have code which builds an array from the samples coming from arecord, then when the array is full (30 sec) I cut a a piece (n bytes) of the array at the beginning and add the chunk (n bytes) at the end of the array so the array keeps the same size. I am now going to code the part that When I want to start recording will pipe the pieces cut at the beginning of the array to lame which will code them into an mp3 file.

Thanks for the advice