Let's consider this very nice and easy to use remux sample by horgh.

I'd like to achieve the same task: convert an RTSP H264 encoded stream to a fragmented MP4 stream. This code does exactly this task.

However I don't want to write the mp4 onto disk at all, but I need to get a byte buffer or array in C with the contents that would normally written to disk.

How is that achievable? This sample uses vs_open_output to define the output format and this function needs an output url.

If I would get rid of outputting the contents to disk, how shall I modify this code? Or there might be better alternatives as well, those are also welcomed.

Update:

As szatmary recommended, I have checked his example link.

However as I stated in the question I need the output as buffer instead of a file. This example demonstrates nicely how can I read my custom source and give it to ffmpeg.

What I need is how can open the input as standard (with avformat_open_input) then do my custom modification with the packets and then instead writing to file, write to a buffer.

What have I tried?

Based on szatmary's example I created some buffers and initialization:

uint8_t *buffer;
buffer = (uint8_t *)av_malloc(4096);

format_ctx = avformat_alloc_context();
format_ctx->pb = avio_alloc_context(
                buffer, 4096, // internal buffer and its size
                1,            // write flag (1=true, 0=false)
                opaque,  // user data, will be passed to our callback functions
                0,      // no read
                &IOWriteFunc,
                &IOSeekFunc
);
format_ctx->flags |= AVFMT_FLAG_CUSTOM_IO; 

AVOutputFormat * const output_format = av_guess_format("mp4", NULL, NULL);
format_ctx->oformat = output_format;

avformat_alloc_output_context2(&format_ctx, output_format,
            NULL, NULL)

Then of course I have created 'IOWriteFunc' and 'IOSeekFunc':

static int IOWriteFunc(void *opaque, uint8_t *buf, int buf_size) {
    printf("Bytes read: %d\n", buf_size);
    int len = buf_size;
    return (int)len;
}

static int64_t IOSeekFunc (void *opaque, int64_t offset, int whence) {
    switch(whence){
        case SEEK_SET:
            return 1;
            break;
        case SEEK_CUR:
            return 1;
            break;
        case SEEK_END:
            return 1;
            break;
        case AVSEEK_SIZE:
            return 4096;
            break;
        default:
           return -1;
    }
    return 1;
}

Then I need to write the header to the output buffer, and the expected behaviour here is to print "Bytes read: x":

AVDictionary * opts = NULL;
av_dict_set(&opts, "movflags", "frag_keyframe+empty_moov", 0);
av_dict_set_int(&opts, "flush_packets", 1, 0);

avformat_write_header(output->format_ctx, &opts)

In the last line during execution, it always runs into segfault, here is the backtrace:

#0  0x00007ffff7a6ee30 in  () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#1  0x00007ffff7a98189 in avformat_init_output () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
#2  0x00007ffff7a98ca5 in avformat_write_header () at /usr/lib/x86_64-linux-gnu/libavformat.so.57
...

The hard thing for me with the example is that it uses avformat_open_input.

However there is no such thing for the output (no avformat_open_ouput).

Update2:

I have found another example for reading: doc/examples/avio_reading.c.

There are mentions of a similar example for writing (avio_writing.c), but ffmpeg does not have this available (at least in my google search).

Is this task really this hard to solve? standard rtsp input to custom avio?

Fortunately ffmpeg.org is down. Great.