What we've gone over. Hopefully by now you have a good concept of how time is used in the Media Kit. As we get into lunch, we get into questions. If you don't understand that, come talk to us, the difference between realtime, media time and performance time and the role that latency plays in the relationships.
Hopefully you understand the Media Roster is used to control everything and that is what you use it to do.
Let's go ahead and look at the interactions between applications and nodes. So, in general, what happens when you tell the media roster to start a node, what node functions are called, you know. Basically how the transactions happen behind that so everyone has an understanding as we go into the afternoon where we are.
Application and node interaction.
Next, kind of a little more detail. This is how it breaks down.
Browsing nodes, creating them.
This is kind of the order that an application would generally tend to do things. You start an application from scratch, one should do something, it will generally follow the steps. Browse through the nodes and create the ones it wants. Set the time sources for them, find input and outputs. Hook them together. It's going to pre-roll and start.
And then we're going to talk a little bit about buffer flow. Applications don't have anything to do with the buffer flow. They're sitting back and watching things happen. We want to make sure everyone understands what is happening in the background when buffers are flowing.
We're going to look at a little bit of the seek operation. You can bring up parameters which you might change the buffer flow by changing the size of the window that the video is being displayed into and you need to take account of that.
We'll talk about tearing down the chain. Go through these fairly quickly because this is the meat of what we will cover in the afternoon.
Let's look at browsing nodes. The application looks for a node that can handle a particular kind of data. It will ask the Media Roster to find a node that handles X, Y or Z. Find a node that knows how to read QuickTime files. That is going to result in some communication between the roster, the media server, add-on server to find the proper media add-on to ask it do you handle this.
So the add-on is going to report all the different flavors of nodes that it can create.
Here's a quick look at functions. For example, you can call BMediaRoster::GetDormantNodes, and for each node that returns, you would call BMediaRoster::GetDormantFlavorInfo to actually see what happens.
So when you actually start iterating through the flavor info, the media add-on is going to have those basic functions.
This is the model we're working with. I won't spend a lot describing these. We will look at the concept, what it does. We will flip through a slide and show you kind of the functions that happen in the background, but I'm not going to mention each of them. If you have a question specifically about one of them, hopefully they're fairly obvious, but if you don't understand some of this, ask me and we'll go back and cover it.
That's what happens when you browse nodes. When you create nodes, basically what you are doing is the application has found a node that it wants to create. It is going to ask the Media Roster to create that node for me. That is one model and that's fairly straightforward. We talked about that. You then -- you ask and it gets created.
What happens in that case, the media add-on looks at the flavor it has been asked for and presents it. The other thing that can happen here is the application can have a node in and of itself that it creates, and the correlating thing is it will register that node with the media server so other people can find out that info.
The functions in the back end are pretty straightforward. You are telling the media roster to InstaniateDormantNode. It will tell that add-on instantiate for me.
If you create it yourself, the application creates it itself. It is responsible for actually performing the registration operation.
Setting time source. Fairly straightforward. The application chooses a time source for the master clock of the chain, and the node, when it gets time source called on it, does whatever it has to do to make sure it can sync to that time source.
As we mentioned earlier, if you have a video card that has a concept of time, and you have an audio card that has another concept of time, and the application -- so both of those nodes are most likely to have a time source associated with it, so it could be used to sync the chain. But, generally, only one of them is going to be used.
If someone chooses -- if the application says I want to use the default time source and that default time source happens to be the audio card and it calls SetTimeSource on the video card, that video card needs to make sure, when it reports times and stamps buffers, it is reporting times and stamping buffers in the time that its time source thinks it is. And if that means it has to do translations, it has to do translations. SetTimeSource means do whatever you have to do to make sure you stay in sync there.
Pretty straightforward. SetTimeSource. SetTimeSource. Time warp is basically a notification that the node is given saying you've had a new time source set to you, your sense of time is different now. Do whatever you have to do to account for the fact it might not be the time you thought it was.
So after you got everything set up and the time source is set, you want to loop through the inputs and outputs of the various nodes to hook them together. Pretty easy. Will probably look for free input and output if it wants to connect them. And the node will respond by reporting information about all of its outputs to all of its free outputs or inputs.
Again, functions called for here are pretty straightforward. Inputs belong to consumers. When you ask -- tell the media roster get the free inputs for this consumer, it will call the consumer function and inputs and iterate through them. And correspondingly on the producer's side, you deal with outputs. Inputs for consumers, outputs for producers.
Now you've gotten everything. You have all your nodes, you've found all your appropriate inputs and outputs, you want to hook them together.
So you connect them, and when you connect -- when an application connects two nodes, it specifies the source and destination that it wants to deal with for those inputs and outputs.
And it says here's the format. I'd like to make the connection now. This is what I would like to do, to hook them together in this format. That format may or may not be fully specified. As the example I gave earlier, all the person cares is the connection is raw audio. It lets the nodes negotiate what sort of raw audio I'm going to use here. That is the negotiation phase that fleshes out all the details of the format so they know exactly what the data that is being passed to them looks like. If the negotiation is successful, the connection is made and the application is happy. If it is not, the connection fails and the application is informed that it didn't work.
So the functions for doing that, a lot of it happens in the background. You ask the media roster to connect. It will ask the producer first, please, I'd like to connect you to this format, does this format work for you, and fill in any bits and pieces you care about.
Assuming that format is acceptable to the producer, it will then ask the consumer, okay, here's the format the producer would like to connect to, does this work for you, fill in any bits and pieces that are still left undeclared.
At that point in time, it will go back to the producer, okay, everything looks good. Reserve a connection for this other person. It will go to the consumer, all right, everything looks good. Do you still want to connect. And if it returns positively, the connection is established.
Audience Member: Is there a race condition between asking for the free inputs and outputs for the connection?
Stephen Beaulieu: Yes, there can be. What you do is you find something that -- the reservation of the input or output doesn't happen until the connection process starts, so you could find that you want to get an output.
From the application side, it says I'd like a free output from here and a free input from there, and by the time it gets around to do that, another application has done the same thing. If it happens, it fails and you start the process over.
But some nodes only have one input and one output. Some nodes can only handle one thing at a time. If you are in that situation, you're stuck. Someone else got it -- got it first.
I do not believe there is a mechanism so that an application can say, no, no, no, don't give this to anyone else but me, I'm going to connect to it, mainly because the connection -- the format it might want to connect might not work.
I'm not sure if we're going to look into that sort of thing. I know it is not there. An application can't reserve something now. It will have to see if it is available at connection time.
Other nodes like the mixer can take any number of inputs in. There is always going to be a free input. The only reason the input would fail at that point is if they can't negotiate the format correctly or you run out of memory.
It will depend upon the node. But, yes, there is a race connection meaning that you might not be successful because the input is no longer there.
Audience Member: I didn't understand in the -- you have contacted a node and trying -- through its inputs. If the inputs are distinguished in some way, if is there some way of telling which sort of input you've reached.
Like in a lot of audio applications there will be two inputs: A carrier and a modulator. And you want to make sure you hook up the right thing to the carrier and the modulator, but the sequence, it looks like they're both inputs.
Stephen Beaulieu: That would be each -when you get an input, remember an input is a complete description of the format. If you are asking for an input, what you are going to get is the node that you are dealing with, you are going to get the source which will be set to null because it is not hooked, the destination, but you will also get a format.
If that input only accepts things of a given sort of format, and it could be that the fact that it is a carrier or something like that, something that could be specified in the format, I don't know. Then you would not find something you want to connect to.
And you could look at the formats that are returned to see if that is the connection. Basically there will be an iterative process. I've found a free input or output and here's the information. And you can look at the format, and if it's not what you want, you keep moving on.
Again, that might not -- I'm guessing that doesn't answer your question. I'm not sure. I don't know what you would do under the circumstance where you would have a modulator and that. We can find out for you.
Audience Member: To qualify that. Essentially the two input streams would be exactly the same format since the data going across --
Owen Smith: One thing you could do, each input could have a name.
Audience Member: Because the connections are named?
Owen Smith: Yes. Each input has a name associated with it. Each output has a name associated with it.
Stephen Beaulieu: Which could be used to describe this is this node's modulator. Thank you very much, Owen.
Audience Member: Two questions. First is application inter-tie, which is to say passing these streams between applications. Is that supported? And second, you had mentioned with respect to graph tear-down, are those going to be addressed?
Stephen Beaulieu: In terms of my application, is going -- you have one application which would insert itself in someone else's application stream?
Audience Member: Not necessarily a plug-in. Let's assume I've got a producing application and a consumer application and I want to glue them together and they're not going to the outside world, or maybe it is a third party plug-in.
Stephen Beaulieu: Yes. Basically the way it would work then, the producer application would have a producer node that only existed in and of itself. You would have the consumer application which would have a consumer node which instantiates locally. You could -someone has to hook them together.
Audience Member: Between the applications, the inter-tie is a supported feature?
Stephen Beaulieu: Yes.
Audience Member: The other question has to do with support for multiple time basis to support media slip. One can build slip converters which allows either dropping of frames for the interpolation of audio without audible artifacts, and those are important components in solving real world problems. The need is for support of multiple time bases because the inputs and outputs have different time bases that are reconciled by a skew factor which you have to keep track of. Is that something you folks are going to want to do?
Stephen Beaulieu: That is something that can be done. Basically what you are talking about, there is a sync node. When you send a buffer downstream, that buffer has an identifier of time source that stamps -- that was used to stamp that buffer card.
Audience Member: That is to say, you support multiple clocks. In other words, I can have two different time bases and keep track of the difference between those two by a slip processor that is a node in the system?
Stephen Beaulieu: Yes. What that node could do, that node could be set. Let's take an example. You want to sync audio and video, you need to do some processing beforehand after they're recorded and processing afterward. You want to make sure when they go through this node they're in sync and everything is time-stamped appropriately.
What you could do is that sync node would be set to the time source downstream. And it would say when I actually send buffers I stamp things downstream, but as each buffer comes in, it can see this buffer was stamped with this other time source. You can use that other time source to create another time source. You can find out what that time is and you can make the translation.
Audience Member: Then I can have an asynchronous stream coming in which actually logs the corresponding skew between the two?
Stephen Beaulieu: Yes.
Audience Member: And thereby reconciles time.
Stephen Beaulieu: You could do that, yes. Successfully answering the question is good. I like that. I want to make sure we can do -- ask away. That's fine. We're kind of merging the Q and A at the end with this section.
Audience Member: You were mentioning connection nodes. You are just connecting two nodes. For audio things you have a graph in the end, so you need to connect a number of nodes. Two nodes may connect perfectly in one format that they both support, but in the end may break the chain because the format accepted by these two nodes cannot be chained again.
Stephen Beaulieu: Right.
Audience Member: So isn't there a need for a mechanism that would allow another overall view of the graph to be sure that nodes can be connected even though it is not a perfect format for the two nodes at the beginning of a chain?
Stephen Beaulieu: Yes. You can do that. The way it works is the application then needs to determine -- to do that correctly, the application needs to determine what do I need to do, what are all the nodes I need in the chain. And then it goes through and asks for input and outputs.
Through the inputs and outputs it is going to contain a format that says here's the format I do. Here's a format definition. It can then look at all the format definitions and say, okay, this -- in general, if I've got a node in the chain here that only supports a certain time rate.
Audience Member: Okay. Is it -- an application task to discount things which is a general problem. I mean, you have these kind of negotiations capability in the node, then the graph is not only an application problem, it is a system problem.
Stephen Beaulieu: Okay. There are -- there are two things about that. Right now the application is tasked with connecting the nodes in a format that both of them can accept.
Now, in many cases that format can be wild cards most of the way, but if one node -- like I've got an audio node that will only accept 44K floats. If you try to connect with anything else, it will not work.
So the application then has to connect it to a node that can give it that. And it does that by looking at the format and saying can this node perform it.
It can try to connect them and see if it fails. But if you have a problem where you need to have -- if you have a very long chain and you need the form to be consistent across the application, the application is currently tasked with having to go and in general find a format that will work. Now, it's possible that the nodes themselves are really flexible and that they'll do resampling and do all of that.
Now, the other pieces we're working on are some convenience classes that would basically say I have this file. For example, you want to play a file and the application does not care about the nodes behind it. It just knows I want this file to play. We're working on building systems whereby the actual node work and the connection stuff is handled for you. But those systems aren't here now.
Audience Member: Because there is the problem of nodes not being able to connect because of another one point. But also the problem of efficiency maybe doing a little of the work in the beginning of the chain will help all the other nodes perform more efficiently.
Stephen Beaulieu: Yes.
Audience Member: As I see it, you get binary answers, not qualified answers. I can perform with this format, but I'd rather perform with this format with such a coefficient.
Stephen Beaulieu: Well, one thing that happens in the processing of the nodes is when it can't accept a format, I can't accept that format but I can accept this one. So you can -- we don't have an answer, we don't have a complete easy-to-do answer for this problem. It takes work.
Jeff Bush: You have an inherently complex problem where you have nodes doing this kind of weird combination of things. In the case where you are using those nodes, the application has to know what it is doing.
You could design a system where nodes have this complex messaging system that go up and down to try to find something in common with all of them. But it would make it unnecessarily complex with the node writers. Complex technology stuff. I think you would make the nodes unnecessarily complex if you made the negotiation that way. The way it is now, it's very flexible. You can basically ask a node for everything: Where are all the input and outputs you have? What is the performance? So you've got all the information that you need to build the entire chain.
Generally, that assumes you have a case where the output is not going to be published until you have an input which generally you know all the types that can output. Does it make any sense?
Stephen Beaulieu: In fact one of the information that is available in the node flavor is what format of information I can support. So you can iterate through all the nodes and say here are the formats of the information I can deal with, and you have to look and find them.
So media processing is complex. There is only so much we can do to simplify the matter. Writing a media application is inherently a complex task and there is only so much we can do.
Audience Member: Are there format conversion nodes which ease the format conversion? I think what was being hinted at that there is a combinatorics explosion which occurs if you have constraints and you're exploring the entire space. So if you have format converters, at least you can say so this only speaks that node, that only speaks those nodes and you can at least bail out.
Jeff Bush: Absolutely. That is very easy to do.
Stephen Beaulieu: Format conversion nodes, to answer what Jeff is saying, are good things. Many nodes can do the format conversion themselves. But having people that -- we do not have an audio format conversion node that is built into the system right now. Such a node can be written. All you need to know is how to translate it from one to the other. We should look into providing those.
I grant we should do that. The developers out here who are writing nodes, if we have something like that that is not there, you can also write it. We need to make sure it happens.
Audience Member: It needs to be system-level, though, because it is what solves the connection problem.
Jeff Bush: It does. The formats are registered.
Audience Member: The conversion -- the format converter needs to be a system-level node because its existence is used in the connection solution problem.
Jeff Bush: They're global. You have add-ons.
Audience Member: You're saying the operating system does not -- leaves the application entirely to worry about the format dichotomy?
Stephen Beaulieu: No.
Jeff Bush: That might be a topic for later.
Stephen Beaulieu: Let's talk about that a little bit later. There are systems -- add-ons are global, which means you can access and find out about those things.
I would agree it would probably be a good idea if the Media Roster allowed you to find format conversion nodes to do something. You could say I've got a node that can take any audio in and any audio out and you can register that says I'm a converter node. That would be a good system. We don't have that system right now. That is up for engineering to decide if we need to do that. I agree that something like that would be good, and I'll take it back to engineering. What we have now, which is what we're trying to tell you this is how it works, here's how we're working on it. We're looking to improve this.
If you have a suggestion, which that is very good one, let us know. We might not be handling it yet. It might be on our timeline and it might not. If it needs to be on our timeline, it is good to get the feedback. I would like to take one more and then we can cruise through. We've got like three or four slides here and then break for some more general questions.
Audience Member: Before you mentioned rate, and I'm just curious how rate is communicated to the system in terms of a change when you, like, say play something half as fast. Is that represented as media time or performance time?
Stephen Beaulieu: It is represented as play rate -- it is represented as enumerator and denominator. You play at half of what the format says that you're connected with which says you're supposed to play at. That means to support that node, if it has to produce, the node might have to copy buffers. It gets in a buffer. Play rate is tricky.
Audience Member: Is that communicated down the node chain?
Stephen Beaulieu: No.
Audience Member: If you are playing at half speed.
Stephen Beaulieu: What you would probably do, tell the producer read from the file and produce at half speed. Then it stamps everything -- the format looks the same. It is still sending buffers at the same sort of time rate, it is just what is the data in those buffers.
Jon just gave me a good answer for this. Play rate really matters for the farthest upstream producer because they're the only one that knows that you are reading from the file, you're the only person that knows about media time.
You basically say here's the duration. I'm supposed to be playing that minute one in the file, and I've suddenly been told to produce data half as fast. So it then does whatever work to do the mapping between what performance time equals what media time. Everyone downstream of that, it doesn't care. It's just buffers worth of data coming in at the right format. So that is something that the producers that -- that is a production thing that is transparent down the stream.
So we've gone halfway through our presentation. The rest of these should go fairly quickly, I hope.
Preroll and start is straightforward. Basically after you found everything and you got them connected properly, if you are building a chain from scratch, you want to tell each node to preroll, do whatever you need to do to get ready for starting, and then the application will tell the node to start and then the node starts running.
This afternoon we will go into the issues of if you have a node chain, which nodes do you start first and things like that. That is part of the application track this afternoon.
So when you move on, this is really straightforward. PrerollNode, StartNode.
Onto the next thing. Buffer flow. So here is how it works. The producer farthest upstream asks for a buffer from a buffer group, which is essentially a container of buffers. In the afternoon we will talk about those as needed. That is a node thing.
So the upstream -- the node farthest upstream, the ultimate producer, gets a buffer. It then fills in the header and points the data at it correctly and then it sends it downstream.
The downstream node gets the buffer and does its work on it, and if it's in the middle, passes it on or copies it to another buffer depending on if there is a decode process going on.
Eventually it will get all the way downstream where this particular buffer isn't going to go any further. That downstream node recycles it and sends it back to make it available upstream.
But if you are an application, you don't see any of this. You don't do any of this. You just care that the nodes are doing it correctly. If you are a node, you need to know how to do it correctly, and we'll cover it this afternoon.
The function and node are fairly straightforward. You request a buffer from the buffer group in the node. And there is a sending process. The producer sends the buffer, the buffer receives it. You then tell the buffer to recycle itself.
Seeking. Seeking generally occurs on the -one of the ends of the chain. Like we mentioned earlier, only the ends of the chain, the file producer or file writer know anything about media time. And seeking basically says at this performance time this is what your media time should be and go from there. So the node adjusts the specific media time. And it's, again, straightforward.
There is a seek and then there is a time warp only if what you've just seeked is a time source, which is complicated. But generally will not happen. Generally you shouldn't do things like seek time sources. Generally says it is now 10 years later. And everyone pay attention to that. Generally not something you're going to want to do terribly often.
Parameters. As we said, there are -- nodes can provide information about the configurable parameters. And application can ask for those parameters and ask for several things to be done with them. It could say to the node, would you on your own create a control panel that the user can use to access and configure things. Or it could say, give me a BView that represents all of your information. Or it can say give me all of your controls and I'll create whatever I want.
The node basically just publishes the information, and then the application is responsible for providing the user interface through these classes we have available.
So in the application track this afternoon, we'll talk from the applications side about getting those parameters and how you turn those parameters into a user interface. On the node side we will talk about how you create the parameter web that accurately describes what is configurable about you.
The inside. Basically StartControlPanel to tell the node to run it all by itself. You get web and change values through the SetValue things. Again, we will cover it this afternoon.
Tearing down the chain. Generally when you are finally done, the application is going to stop all the nodes. It needs to stop, disconnect them. If it created in itself, it will delete its node which will unregister it -- release its node which will go ahead and unregister it. The node stops running and it's going to handle its disconnection. And it's finally deleted when no one else actually has a reference to it.
Again, the function's all back in the back. One interesting thing to note about telling a node to stop. You can tell -- an application can tell a node to stop synchronously. I want you to stop immediately and the media roster won't return until that node has stopped until it has done everything it needs to do to stop.
You can also tell a series of nodes to stop at this performance time so you can have the chain stop itself naturally. But in tearing down, sometimes it can be convenient to say I want to make sure you are stopped before I move onto the next one.
Disconnecting. Calls first BBufferProducer::Disconnect and BBufferConsumer::Disconnected. You unregister. When the node is finally released and isn't needed any more and no one has a reference to it, the actual BMediaNode gets deleted.
So that is application and node interaction. Kind of in a nutshell what we're going to be talking about all afternoon. I will take more questions.
It is 11:40. Lunch is going to be served at noon. So we can take 20 minutes worth of questions here. I have some announcements first beforehand. To pick up your beta CDs, there is the registration booth up there.
Let's take questions for the next 15 or 20 minutes.
Audience Member: I have some general questions. What happens when things go bad? Let's say you have an application that registers a node. The application is crowded. That node is in the middle of a stream somewhere. What is going on?
Stephen Beaulieu: Well, crashing is bad.
Jeff Bush: We cleaned that up.
Stephen Beaulieu: Jeff will answer part of that. We do some things in the system to clean up connections for bad nodes that have gone away and to disconnect everything so it doesn't take everyone else down with it. What is the minimum of what will happen? That series of connections won't work any more.
Audience Member: Will at least the client app be notified that things went bad?
Jeff Bush: Yes.
Stephen Beaulieu: Yes.
Audience Member: The client app would set everything up. From a user interface point of view, it is particularly bad when everything crashes.
Stephen Beaulieu: There is a system in nodes for reporting errors. Basically a node -- when the nodes on either side get disconnected, they're informed of that disconnection and they can say my connection went away, and they can report that they're in distress. That bad things have happened.
Audience Member: Another sort of related question. In your debugging of these nodes, is there a leak detection, isolation of memory leaks, that sort of thing? Is that something that can be done easily so at least we can identify which node is the bad player and interstream?
Stephen Beaulieu: Debugging nodes. So if things are going bad, how you determine like which node might be leaking memory and things like that?
In Genki, you're going to have the ability to instantiate a node inside your application. Instead of leaving in the media surface, there is going to be a definition of which nodes need to be instantiated inside the media add-on server for things like hardware nodes and which can and perhaps should be inside your own application.
And then if you need a particular one, you can get it instantiated inside your application. And, you know, if it has been compiled, you can get a debugged conversion from that person and you can step through with our debugger and figure out what is going on.
But there are problems that you run into. To a certain aspect, the node is black-boxed. If the developer doesn't give you the ability to debug it, it doesn't matter what you're going to do. But if you can get access on the other side, you might be able to figure out if they're damaging.
Audience Member: What I'm thinking, if you are writing an app and components are written by thousands of people or hundreds of people, or whatever, if something goes badly, is the consumer going to say your app sucks, or is the consumer going to be able to say Joe Blow wrote a bad node?
Stephen Beaulieu: I do not know.
Jon Watte: You can compare device drivers if you get a device driver for that sound card. Then, you know, once in a blue moon your sound system will crash. But if the user -- how does the user know? It is a hard question. If the user tries another sound card and it does not crash, then it might be the card or the driver.
Audience Member: But that sort of identification isn't in the Media Kit.
Jon Watte: This isn't really a good answer to that. If you have ideas of how to solve that, send me an e-mail.
Audience Member: Protected node modes. We should be able to create a node inside its own team. Watch what one blows bigger or dies.
Jon Watte: Right. We're working exactly on that for Genki. I don't know how complete it will be, but it will be -- we already have a lot of it there.
Audience Member: Could we have a switch which basically says the policy is nodes doing their own protective space? We flip it on for debugging, running a little more overhead, but at least we get the wish that we can figure out which node it is.
Jon Watte: Yes, there could.
Stephen Beaulieu: Back in the back, please.
Audience Member: Concerning the buffer flow, is there a way to select between passing the buffer or is it always a buffer itself, from one node to another?
Stephen Beaulieu: Excuse me.
Audience Member: Getting from one node to another. The data is always copied.
Jon Watte: Data is never copied.
Stephen Beaulieu: The data lives in its own shared memory space to be BBuffer objects. That is the information that is passed from one node to another. Contains a pointer in the shared memory area where the data is contained because we specifically don't want the copies.
Audience Member: Okay. Is there a way to collect, to wait until an operation finishes and then return, or say start the operation and continue on straight until the operation finishes?
Stephen Beaulieu: Basically from the applications standpoint, everything is somewhat asynchronous. We don't have a built-in system that says start this chain and I don't return until this chain is done. However, we do have a system of notifications whereby you can set things up so that nodes will report when they're done doing whatever they are told to do. You can say I want this, the final file writer, you want to know when the file is done being written. You start and you have -- the nodes have a start and stop time. You're not dealing with times that can go faster than realtime or whatever.
When the final file node gets done writing, it can broadcast a message, hey, I've finished. We will -- no way to do it synchronously, but we have a notification system so you know that that operation has occurred.
Audience Member: So I had another question when things go wrong. Mainly, when things don't go terribly wrong. Like when you have -- well, basically soft errors. When you have a situation set up where you have connected everything and reading from a file and you think it is fine when you start up. You find out halfway through that your file is corrupted but you might be able to recover from it.
Is there an error mechanism that allows for both hard and soft errors, where sometimes you need to let the application and the user know that something is not okay, but you think you can recover from it and you might be producing data somewhat wrong?
Stephen Beaulieu: That is a node. If a file reader node knows that its data is corrupted, it could send a message that says I'm in distress.
Now simply sending a message I'm in distress does not mean the node is going to stop what it is doing. If the node can determine, you know, it is totally bad corrupted data and I'm going to crash, it could shut itself down. But probably what will happen, if you get totally corrupted data and you pass it on to a decoder that tries to do it, you're going to crash in those corrupted circumstances.
The idea is how can you detect it? If you can detect it, there is mechanisms for nodes to notify applications. Applications can say I want to start watching for the information for the nodes, when they get -- determine they should send information, basically do a here's information about where I'm at.
Audience Member: So you can use that error message for pretty much any type of message whether or not it is an error or update on data?
Stephen Beaulieu: Yes.
Jon Watte: Not for status.
Stephen Beaulieu: Not for status.
Audience Member: Status is kind of -- it doesn't matter.
Stephen Beaulieu: Okay.
Audience Member: Two short questions. One is can nodes block if they're not receiving inputs so that they don't consume CPU time?
Stephen Beaulieu: Basically that is -- the implementation that we want nodes to do is yes. If they're not doing anything, they're blocked on waiting for the next message to come in. And they don't -- they don't consume processor resources. Of course, whatever memory they've allocated is still around. If it is not ready and if it doesn't have anything to do, it won't run.
Audience Member: The second question is are there any built-in facilities to logically combine several nodes together into a larger supernode that appears as a single node to the user?
Jon Watte: No, but we're working on it.
Stephen Beaulieu: Jon says no, but we're working on something we're roughly calling the node wrangler class which basically encompasses a bunch of nodes. From the application standpoint, it works the same way. From the media node standpoint, it is still a series of different nodes.
Audience Member: One of the problems using nodes, some BBuffer would not recycle. When you call disconnect, nothing was going to be disconnected and the oldest chain just stopped.
Stephen Beaulieu: Yes.
Audience Member: There is no error or time out?
Stephen Beaulieu: Part of the problem is that if nodes are poorly behaved and they don't recycle their buffers, you will deadlock. People will wait on -- I need another buffer. And that is a poorly-implemented node.
Jon Watte: We can't know whether the node is actually using the buffer or not done with it.
Stephen Beaulieu: Yeah. Jon said we don't know when a buffer has not been recycled.
Audience Member: When you call delete of your buffer group, it calls --
Jeff Bush: If somebody were still using the buffer, you will delete an area that is still being written to. You can't know because it hasn't quite finished yet. There is no such thing as a free lunch. If you put all the checking in to make it super stable, it would be slow. That is the trade-off you make. In this case, your application is going to stop working because you have a bad node, but generally things can recover from there as far as the system is concerned.
Audience Member: The memory leak problem is a recurring thing in C programming. People have addressed it with things like garbage collectors.
I think there is a tear-down and memory leak invitation here. I think some serious thought as to either tools for fleshing it out or an adjustment that takes on the responsibilities of garbage collection is important, otherwise your system gets delicate.
Stephen Beaulieu: There are some issues. We have some debugging tools. One of them is you can open up a terminal, type a certain value and you can determine whether people are writing over bad memory. We're also working with the people with Pavel, one of our application engineers. We had for our 3, but not for our 4, a leak-checking group. We're working on getting it implemented so you can specifically turn on leak-checking. And when your application is done, you can see a list of here's the memory that was leaked and here's a call stack. But it is a node and you don't have information for like what those numbers here -- this function was here. This bit of memory was what was called into the function. You don't know what that is, but you might be able to determine which node did that.
Jeff Bush: That is not going to solve that.
Jon Watte: I should probably introduce myself at this point. My name is Jon Watte. I'm the chief scapegoat for the Media Kit. I've been the disembodied voice down here.
Another question. You were saying reference counts are known to be problematic. Yes, they are in some circumstances. We only use them to check which application is holding onto or one as to which node. When an application goes away, we are supposed to release all the reference counts held by that application. There has been bugs in that implementation. Genki is supposed to fix those.
So in this case, I think that reference counting nodes and then releasing those reference counts for the app, if the app goes away unexpectedly or not, will solve the problem of when those nodes live and when those nodes get collected. Did that answer your question?
Audience Member: Do you keep an individual reference count for each app and a composite of all the counts, so when an app dies it takes its particular count with it?
Jon Watte: Correct.
Audience Member: And it doesn't corrupt the count of other counts?
Jon Watte: That is correct.
Audience Member: I've been using the Media Kit for video, and the problem is the BBuffer which is not copied between the nodes that is perfect, but once you try to display it into a view because I'm using no windows due to the replicant technology, we have to copy it into a BBitmap. I was wondering because both BBitmap and the buffer have a shared area, can we have a constructor as a BBuffer?
Stephen Beaulieu: That's a good question.
Jeff Bush: The media player does exactly that. If you create a bitmap with the continuous flag, it is created in its own area. That's important. It has to be in its own area because the app server needs to move it around. If you clone it, it can't do that.
If you create and confine the area for it, when you -- you can actually create buffers from that and it will actually write into that bitmap. If you want to get even more extreme, there is support and it has not been totally -- nobody has used it yet, but there is support for actually passing rectangles back to the node.
Stephen Beaulieu: There are two things. In the sample code we're providing you, it has a bitmap buffer class, which is subclass of BBitmap, that when you create it will -- you can have a buffer function that will give you a buffer on that. The problem that the -- the problem we have right now with the copying is that currently it does not support the app server for creating bitmaps that use this area. So if both the producer, the ultimate producer and the ultimate consumer need to have bit maps, you have to have a copy right now.
We're looking into whether we can, and if we should, support the fact that you can create a BBitmap from that area in which case we wouldn't have any copying even if you are using views on both sides. That is something we're looking into.
Audience Member: I'm wondering how nodes map to threads. Always one thread per node?
Stephen Beaulieu: Come to the node session this afternoon. In general, there needs to be -- a node has a control port that gets messages. There needs to be a thread that reads from that port.
And for independent nodes that are by themselves, for example, the model we're suggesting, it has one thread that handles that and does all the work. But a node doesn't have to. For example, Quake could want to use the Media Kit for delivering audio, but Quake in its own threading system, it doesn't want to communicate.
We're working on building a set of nodes that you provide the thread for. You call a function on this class that will then do all the node work and you need to promise to call it in the right time. In that case, the node doesn't have a thread. It uses the thread of the application. So for most of the circumstances, nodes will most likely have one main thread they're dealing with, but they don't have to.
Audience Member: Thanks.
Stephen Beaulieu: Chris.
Audience Member:: The problem of misbehaving nodes is going to be troubling to people. It seems like we should probably have a standard test harness for putting nodes that are consumers into a system of known good producer for that node and saying is it behaving properly, and vice versa for intermediate nodes and producer nodes.
Stephen Beaulieu: I will take it as a request for DTS to look in to doing that. And we will see when I can allocate resources for it. But I agree. This will be the last question because it will be lunchtime in a minute.
Audience Member: I have a particular problem where I was trying to find a range of values that a node could use. Like, for instance, it can do 15-bit video and 32-bit video, but none of the other formats. In the media format you can specify a wildcard. I can take anything, or a specific value, which means I can take this specific value. Is there a way of when using the -- the kit mechanism to say -- I can provide an input that can do 15-bit and 32-bit video but nothing else?
Jon Watte: You can publish two inputs, one that takes 15-bit.
Audience Member: But if you can only do one input, you can't handle two.
Jon Watte: Yeah, when one gets connected, the other goes away. They're not persistent. Now, if you can do 15 or 32-bits and you can do 30 or 29 frames per second and you can do 320 by 240 or 704 by 488, then you start getting exponential growth.
And then at some point you have to decide whether it can do everything and deny connections or formats that actually you can't do.
We looked into ranges, but the API got really hairy with multiple indirected pointers, and we just decided that since people are either probably going to be well behaved and trying to do a large subset of all the possible values or they are going to focus on the one thing or two things they do, then this would be sufficient, we hope.
Stephen Beaulieu: Thank you very much.