Be Newsletter

Volume III, Issue 10, March 10, 1999

Table of Contents

• BE ENGINEERING INSIGHTS: BeIDE Add-on API, By John Dance
• BE ENGINEERING INSIGHTS: Automating Myself into a New Line of Work, By Steven Black
• DEVELOPERS' WORKSHOP: Media Kit Basics: Build Your Own Mixer For Fun And Profit!, By Owen Smith
• The First Media Kit Conference, By Jean-Louis Gassée

A week or so ago, I needed to look into a program that was partially generated by the tools bison and flex (yacc/lex). The program was built with a makefile, since the BeIDE can't handle bison/flex as part of the build...or can it? I figured that since gcc and mwcc are actually handled as add-ons in the BeIDE, you should be able to handle the limited needs of bison/flex the same way.

The BeIDE comes with a shell tool add-on that lets you specify shell scripts to be run during steps of the build. This could actually be used for running bison/flex, but the shell script would be executed on every build. Since flex and bison produce files that I'll use as part of my source, I didn't want them running on every build. I could write the shell script to check whether flex/bison actually need to run, but it would still be less than optimal...and besides, it wouldn't give me enough to write about in this article.

As I was thinking about writing an add-on for flex and then for bison, I realized that I didn't want to write multiple add-ons at all. In fact, I wanted an add-on that would work for both flex and bison, and that could also be easily extended for mwbres, rez, or a host of other tools of this sort. But how to do this? The BeIDE add-on architecture allows a single add-on to specify more than one builder. Given that, I decided that I should be able to factor out a minimal amount of specific code for each tool. I went ahead and built a simple generic translator add-on that I then modified to handle these multiple tools. It can also be easily extended as I encounter more tools of these types that I would like to use.

The BeIDE supports editor and builder add-ons. I'll leave the editor add-on architecture for another article. The generic translator project can be found at ftp://ftp.be.com/pub/samples/beide/translator.zip. It would be more instructive to look at the source as you're reading the article. Unfortunately, R4 did not include the headers needed for writing BeIDE add-ons. Until you get your copy of R4.1, the headers and textual documentation for the add-on API can be found at ftp://ftp.be.com/pub/samples/beide/Plugin_API.zip.

As the BeIDE starts up, it looks at all the add-ons in a known location and calls two important functions:

extern "C"
status_t MakePlugInView(int32 inIndex, BRect inRect, MPlugInPrefsView*&
outView)

extern "C"
status_t MakePlugInBuilder(int32 inIndex, MPlugInBuilder*& outBuilder)

These two functions return the MPlugInBuilder and MPlugInPrefsView that do the work and communicate with the BeIDE. The MPlugInBuilder is a wrapper for the external tool. It is used to specify what tool will be run, at what stage of the build the tool is used, what parameters to pass to the tool, and finally, how to interpret output from the tool. The MPlugInPrefsView provides the user interface for getting and setting options for running the tool. In some cases an add-on will provide multiple MPlugInPrefsViews.

In the functions above, the inIndex parameter allows me to create more than one view and more than one builder. Both functions will be called with an incrementing inIndex until they return something other than B_NO_ERROR. To make this example simple, I don't even support preference settings. That will be left as an exercise for the reader. Don't you hate it when authors write that? Now I know why they do. They don't want to do the work themselves.

Writing an MPlugInBuilder is a straightforward process. (especially when you don't worry about user preferences). There are about 16 member functions that must be overridden to create a concrete MPlugInBuilder class. Many of them can be written in just a few lines of code. Remember that I decided my add-on would support a variety of translation tools. This means that I need some way to customize a shared body of code for each individual builder. We are using C++, right? Then why not a class? The main work of the MPlugInBuilder is handled by GenericBuilder, which does as much work as possible before delegating specific needs to a BuildHelper it owns. MakePlugInBuilder wants to create a MPlugInBuilder object; I create a GenericBuilder in each case but pass each one a specific helper class. Here is my MakePlugInBuilder:

extern "C"
status_t
MakePlugInBuilder(int32 inIndex, MPlugInBuilder*& outBuilder)
{
long result;

switch (inIndex) {
case 0:
outBuilder = new GenericBuilder(new FlexHelper);
result = B_OK;
break;
case 1:
outBuilder = new GenericBuilder(new BisonHelper);
result = B_OK;
break;
case 2:
outBuilder = new GenericBuilder(new ResHelper);
result = B_OK;
break;
default:
result = B_ERROR;
break;
}
return result;
}

I support three tools: flex, bison, and mwbres. But you can easily see how to extend this to other tools by writing a new BuildHelper and adding an additional case. To get a feel for writing an MPlugInBuilder, it would be best to go look at GenericBuilder.cpp.

The concrete BuildHelper classes (FlexHelper, BisonHelper, and ResHelper) override six member functions that deal with actions or information specific to each individual tool. I'll talk about just three of these methods here.

One of the main jobs of the MPlugInBuilder is to create the argument list for running the tool. If I had a MPlugInViewer, this is where I would retrieve and interpret the options in the BMessage that was created with the MPlugInViewer. The BeIDE asks the add-on to fill in a BList with a char* for each option. The Plugin_API documentation says that each string must be copied, since the BList adopts them and later will free them. Since free is used, I use strdup rather than new char[].

status_t FlexHelper::BuildArgv(BList &inArgv, const char *filePath)
{
BString outputName;
this->MakeOutputFileName(filePath, outputName);
outputName.Prepend("-o");
inArgv.AddItem(strdup(outputName.String()));
inArgv.AddItem(strdup(filePath));
return B_OK;
}

Notice that BuildArgv uses another helper method, MakeOutputFileName. Each BuildHelper class creates an output file name based on its input file name, with a different extension. For example:

flex: .l -> .cpp
bison: .y -> .cpp
mwbres: .r -> .rsrc

The new class BString can really be helpful for string manipulations. To create the output file we just take the input file and replace the extension:

FlexHelper::MakeOutputFileName(const char *filePath, BString &
outputName)
{
outputName = filePath;
outputName.ReplaceLast(".l", ".cpp");
}

What happens when tools have errors? All output is captured by the BeIDE and then sent to the add-on by calling MPlugInBuilder::ParseMessageText(const char *inText, BList &outList). ParseMessageText has the job of figuring out what is in the message and creating ErrorMessages that are then added to outList. Any ErrorMessages in outList, will be displayed in the Message Window. It's very simple to create "text only" error messages. Just take each line of text and stuff it in a new ErrorMessage. However, it's much nicer to communicate the file and line information so that the user can navigate from that error message to the spot in their source with the problem.

Looking at the type of tools that I wanted to support, I saw they had a similar style for error messages. Here is an error from flex, bison, and mwbres:

        "test.l", line 123: EOF encountered inside an action
("test.y", line 84) error: invalid $ value
File test.r Line 6 # parse error near 'include'

Since I didn't want to write N different parsers, I had GenericBuilder::ParseMessageText be in charge of breaking the message up into individual lines and passing each individual line to the BuildHelper. (If a tool has multiple line errors, GenericBuilder::ParseMessageText will have to be overridden.) The base class BuildHelper provides a few methods to parse lines that follow a format like this:

something <filename> something <linenumber>
something message

(Here I am talking about flex and bison and I write the grammar like that...sheesh!) This means that the different concrete BuildHelper classes can implement the parsing in one line. All they have to do is specify the tokens (the "something" stuff) before and after the file name and line number.

Once the add-on is built, how do you use it? The add-on needs to live in the BeIDE directory under plugins/Prefs_add_ons. Once installed, use the Settings/Target preferences to connect file types and extensions to the tools. To build my project that used both flex and bison I added the following targets:

Extension: l
Tool: flex
Flags: Precompile Stage

Extension: y
Tool: bison
Flags: Precompile Stage

I then added the foo.yy.l and foo.tab.y files to the project. Since flex and bison are used to produce files that are then compiled as part of the project, I also added the files they produce: foo.yy.cpp and foo.tab.cpp to the project. The BeIDE has now been successfully enhanced to work with flex and bison. Once I wrote and debugged the GenericBuilder and FlexHelper, it took me about 5 minutes to write and test the BisonHelper and ResHelper. I'm sure many developers will think up lots of creative uses for the BeIDE add-on API.

My father used to tell me stories about how he tried to create systems that took care of themselves so well that no human intervention was needed. He's had at least one job that, according to him, he succeeded getting to run itself so well that he had nothing to do and left out of boredom.

I try to do the same, although even when I think I have something taking care of itself, there are always more changes to fold in, new features to test, new conditions to take into account. So far, I can't foresee any time in the future when what I do could get boring.

Baron Arnold told you that a newer, better release of the InputRecorder would be available in March with my newsletter article http://www.be.com/aboutbe/benewsletter/volume_III/Issue4.html#Insight2. This is so. He also mentioned that it would contain source code. This is also true.

The source has been cleaned up a lot since last month, along with the user interface. A memory leak in the input filter was fixed. And now you may even be able to run multiple recorders at the same time, though that hasn't really been tested much.

While the original author used BResources to store the record data, I chose another format -- a simple, plain file structure. I implemented all three forms in my MessageWriter class, so you can explore and play with all the options. (The attribute- and resource-based methods need to be modified slightly and recompiled, but they are implemented.)

I decided not to use attributes to store the input records for several reasons. The first one is that I don't really need to, since I don't need to randomly locate input events in the file (I treat it as a simple stream). The second is that there's currently no easy way to tell how large a file's attributes are (with InputRecords the files can be large). Finally, a simpler file format is easier to transfer. (Zip files are limited to 64k of attributes per file, which can easily be exceeded when recording input. The Zap format doesn't have this restriction, but isn't as common.)

A reminder: We do not recommend using BResources as general purpose data storage; we recommend using attributes instead. BResources were designed specifically for storing application resources.

One reason for the delay in getting the InputRecorder (especially as source) to you has been that some input server-related parts aren't quite finished. Some device and filter elements may be changing, breaking compatibility, at some future date, in order to improve them. Some of these things may still be subject to change, although I've convinced Those On High that it would be great let developers have access to the InputRecorder.

The source this week is amazingly straightforward. The headers for the input filter stuff are in /boot/develop/headers/be/add-ons/input_server/*.h, where there are three files: InputServerDevice.h, InputServerFilter.h, and InputServerMethod.h.

Input scripting uses only two of the three input_server add-ons. It uses a device to write input messages to the input_server, and a filter to read input_messages from the input_server. Both use ports to talk to the user-level program.

It should be clear to some of you that the design of the InputRecorderDevice is such that it will pass along any message its port receives directly to the input_server. This means, for example, that you could create plug-ins for your favorite scripting language that let you arbitrarily script input.

Here in the QA Lab we use a simple, straightforward little tool to do just that -- I'll tell you about it in a later article. It contains within it the details of the kinds of messages devices can send. It contains many arcane facts that in some cases I just guessed about, based on how our devices appear do it and what did and didn't work. That's why it needs to be looked at carefully before it's released, to ensure that all the information it contains is accurate.

As an example, one of my favorite scripts that use this tool is called 'normalactions' because it contains things most everyone does at the same time. It utilizes four workspaces: one to send/receive e-mail to/from itself; one to compile and run the sample code; one to browse the web with NetPositive; and one to periodically run audio, which continues playing while everything else is going on. It uses a couple of helper programs to do assist in the synchronization issues -- things like ensuring that the active application is what I expect it to be.

So you see, the long trek toward my dream of a fully automated QA, where all the QA engineers sit back drinking coffee and watching subprocesses kill machines and fetch engineers is on the way. Not in the near future, but it's coming slowly. Maybe one day QA will become so boring that I'll start looking for a new line of work... though somehow that doesn't seem likely.

"Developers' Workshop" is a weekly feature that provides answers to our developers' questions, or topic requests. To submit a question, visit

[http://www.be.com/developers/suggestion_box.html].

I heard a rapidly approaching Canyonero, the deadliest of sport utility vehicles, in the parking lot yesterday, and once again recognized the sound of doom. And sad to say, I was a convenient target for it.

"You know, those audio sample apps you're coming up with are pretty lame," spat Morgan le Be, BeOS hacker and mistress of the black arts, bringing her smoking 'Ero to a halt against the red curb. "Who plays only one sound at a time these days?"

"Well, you can hook up your sounds to the system mixer..."

"Uh-uh, buster -- I want a mixer, just like the system mixer, but for my own private use."

The gauntlet was down, and for the first time, I knew how to parry Ms. le Be's attack. Just a few minutes' explanation, and she was satisfied.

And so I say to you, O, True Believers: Heed the dark magick of the Media Kit. Somewhere in the bowels of the Media Add-On Server dwells a dormant mixer node, as awesome in its slumber as a hibernating polar bear, waiting for you to harness its power.

Me and Sir Mix-A-Lot

This week's sample code shows how you, too, can build a mixer. It's fast, easy, and KitchenAid approved. However, please note that the mixer add-on will not support this until R4.1, though the techniques I demonstrate are indispensable to any application that has to deal with media add-ons. Your URL for today is

Mix-A-Lot is a simple app that borrows the sound playback mechanism from SoundCapture to play your super-phat grooves. Unlike SoundCapture, this one constantly loops your favorite sounds and allows you to mix up to four sounds simultaneously.

How do you use it? Simple: fire it up and drag a sound file from the Tracker into one of the icon docks at the top of the Mix-A-Lot window. The sound will start promptly, and continue to play in a loop until you drag it off the dock. A new sound dragged onto an already occupied dock replaces the old sound.

The classes MixerManager and Channel do most of the important work in this app. MixerManager is responsible for instantiating the mixer and hooking it up. Channel represents an input connection to the mixer, and is responsible for instantiating input sources and connecting them to the mixer.

You'll also notice the standard set of mixer controls in the Mix-A-Lot window. If you paid attention during last week's Workshop, you'll already have a good idea where those controls come from. If not, revisit Mr. Tate, and be enlightened:

Adding It On

The magic behind this mixer madness is the Media Roster's ability to instantiate media add-ons for you. Media add-ons are code modules that can provide media services to any BeOS application. You can find them in two locations: /boot/beos/system/add-ons/media/ and /boot/home/config/add-ons/media/.

Here are some facts about Media Add-Ons:

• Media add-ons load dynamically; like any BeOS add-on, they're loaded when they are needed. The application doing the loading in this case is an important part of the BeOS media system, called -- appropriately enough -- the Media Add-On Server.
• Media add-ons know how to instantiate media nodes, and support an interface to do so when the Media Add-On Server demands it.
• Media add-ons support an interface that tells the Media Add-On Server what kinds of media nodes they can instantiate. Many media add-ons deal with only one kind of node -- for example, the BeOS mixer add-on only supports the mixer node. However, a media add-on can support as many different kinds of media nodes as it wishes. For example, you might want to write a codec add-on that contains both an encoder and a decoder node for a particular format.
• Media add-ons generally have one purpose: to instantiate media nodes that any application can use, without the application having to know any of the messy details of the node's implementation. For this reason, media add-ons are generally self-contained. For instance, there is a Video Window node coming in R4.1 that, when it's instantiated, creates its own window and view to display the video that it receives. All you need to do to take advantage of this node is hook it up to an appropriate video source -- the Video Window takes care of the rest.

Mixing It Up

As I hinted at above, the BeOS system mixer node is provided by none other than the Mixer Media Add-On. But there's nothing to stop you from getting access to the mixer add-on as well and instantiating your own mixer. That's exactly what Mix-A-Lot does.

To find the system mixer, you need to know what kinds of media nodes are at your disposal. This information is provided by objects called dormant media nodes. You can query the Media Roster to get a list of dormant nodes that are available to the system and specify any of the following criteria for your query:

• whether it supports a particular media format in its outputs or inputs;
• whether it matches a particular name;
• whether it matches a particular node kind.

The last criterion is the one we'll use. The "node kind" is a set of flags that a media node uses to inform the Media Roster about its particular characteristics. There are several predefined kinds that help to categorize the nodes you'll encounter -- for example, a node whose kind includes B_TIME_SOURCE derives from BTimeSource, and a node whose kind includes B_PHYSICAL_OUTPUT represents a physical output of your system, like audio output. Conveniently enough, there is also a type defined for the mixer node: B_SYSTEM_MIXER. (The kind of this particular node is somewhat misleading, because it applies to any instance of the audio mixer node, not just the system mixer that you access via the Audio preference panel.)

Once we know the kind we're looking for, finding the mixer node is trivial, especially since there's exactly one add-on right now that fits our criterion. BMediaRoster::GetDormantNodes() is the function that does the work for us. It returns a list of information about each dormant node it finds; this information is encapsulated in a structure called dormant_node_info.

status_t err;
dormant_node_info mixer_dormant_info;
int32 mixer_count = 1;
err = BMediaRoster::Roster()->GetDormantNodes(
    &mixer_dormant_info, &mixer_count, 0, 0, 0,
    B_SYSTEM_MIXER, 0);

What do we do with this dormant_node_info? Well, we can use it to get more information about the dormant node (called the flavor_info), or we can use it to create an instance of that particular dormant node. We'll do the latter through the function BMediaRoster::InstantiateDormantNode(), which takes a dormant_node_info and gives us the media_node that's been instantiated:

media_node mixer_node;
if (err == B_OK) {
    err = BMediaRoster::Roster()->InstantiateDormantNode(
        mixer_dormant_info, &mixer_node);
}

Screaming in Digital

The only thing you need to know from here is how to hook the mixer up.

The mixer node supports an arbitrary number of inputs -- this means that you can always get as many free inputs for the mixer as you want, via BMediaRoster::GetFreeInputsFor(). Here's what you should know about mixer inputs:

• Each input's media format must be B_MEDIA_RAW_AUDIO.
• Each input must be 1 or 2 channels.
• For best performance results, leave each input's buffer size and byte order unspecified; the mixer will set them to their optimal values.

The mixer node supports exactly one output, which provides -- surprise! -- the audio mix. For the system mixer, this output is usually plugged straight into the audio output node. For Mix-A-Lot, we take the output of our audio mixer and plug it into the system mixer. The salient features of the mixer output are these:

• The output's media format is B_MEDIA_RAW_AUDIO.
• The output's raw audio format must be B_MEDIA_SHORT or B_MEDIA_FLOAT.
• The output must be 1 or 2 channels.
• The output must be host-endian.
• Again, for best performance results, leave the output's buffer size unspecified.

I leave you with this challenge: How about adding stop and play buttons for better playback control? Or perhaps you can evolve this modest looper into a latter day Echoplex? Go forth and twist Mix-A-Lot to your own sordid ends! Or, as my childhood hero was fond of exclaiming: "C'mon Flash, let's go catch them Duke boys!"

About a year ago, at our developers' conference in Santa Clara, we formally unveiled the first version of the BeOS for the Intel architecture. This year, on April 9-10, we'll have our first Media Kit conference -- an even more momentous event than our 1998 entry into the world of x-86 hardware.

Not that we didn't enjoy the occasion. For me, developer conferences are always a happy time. I get to meet the people who, in a very real sense, "make" the platform. The people who commit an incredible amount of time, energy, and creativity to our product. We get vigorous feedback, encouragement, and great ideas.

I've often thought that, in our business, you know you've done something right when your platform is perverted. By which I mean -- before other interpretations are suggested -- that programmers or normal people use your product in ways you hadn't thought of. One reason you hadn't thought of them is the myopia you get from seeing your product all the time. Another reason is that developers represent so many different areas of interest and expertise -- whereas, to a hammer everything looks like a nail.

The best part of having your platform "perverted" is seeing that the product is rich enough for others to extract more uses than you'd initially foreseen. For example, the creators of the Macintosh didn't think of it as the engine of what became Desktop Publishing, DTP. The concept had no meaning at the time, just as home computing was ridiculed as a fad. In retrospect it all seems logical and preordained: if you have a great user interface, good fonts, and an easy-to-use laser printer, you'll create DTP.

Ah, but it wasn't so obvious beforehand, and retroactively amused witnesses remember contorted positioning statements suggested by the experts of the day. One of the best was "Graphics Based Business Systems." I trust that I'll be equally amused and humbled someday by a few of my own suggestions for our little company.

So, we had a good time last year, complete with an enthusiastic speech from Intel's Claude L'Eglise and a quasi slip-up from yours truly recounting a recent meeting with Andy Grove. At the time, we had an agreement with Intel to keep their investment in Be confidential -- it was disclosed last November at Comdex. As I proceeded to describe a meeting where I came to thank Andy for his support and to give him a couple of black Be T-shirts -- and photocopies of Intel's stock certificates fresh from the closing one hour earlier, my colleagues grew a little agitated. I realized the blunder I was about to make and abruptly moved to another topic.

This year, we'll reveal the secrets of the Media Kit. OS platforms are sometimes compared to musical instruments, with rendering power as well as expressive power. In our case, to stretch the metaphor a bit, we could say that basic BeOS features such as symmetric multiprocessing, pervasive multithreading, and a 64-bit journaled file system give it rendering power. The Media Kit, on the other hand, gives the BeOS expressive power, the ability to write new music -- that is, new applications that aren't necessarily as easy to write, or perhaps impossible, on other sets of APIs.

We've often referred to the BeOS as the Media OS. The Media Kit is the means by which what was the promise of new OS technology can now be expressed in new applications that address both the creation and the consumption of digital media.

I look forward to meeting you at this conference and to seeing surprising new applications down the road.