GLScube.org Blog

The State of GLScube

It has been sometime that there was any public activity at the GLScube front. The lack of updates were a combination of how some of the developers became too busy, and some other points that I would get to next. But first, let me tell you GLScube is far from being anything but alive.

When we have started development on GLScube, we had the one goal of finding a better user experience in organizing and retrieving documents. That simple goal was then broken down into many technical challenges, and we went on to solve each one of them. As the project progressed, however, we came to realize that we were focusing too much on the technicalities, and always favouring technical decisions over usability-related decisions. Here, I will introduce you to those two issues, and why we reached a point were we have almost “stuck”!

Let’s first get to the usability-related issues, since after all, the project was made in the first place to enhance on usability. One of the most important features in GLScube is Virtual Collections, which are analogous to the traditional directories but whose contents are retrieved at runtime based on an associated query. Some Desktop Search engines, like Apple’s Spotlight in Mac OSX, have a similar feature implemented, though only presented to the user as an extra feature he can use in addition to the traditional directories!

This to me is certainly an issue. The concept of convincing users of using both static and dynamic means to organize their documents is utterly ridiculous. It is already hard enough to teach an average person about the concept of hierarchical directories and how he can walk through them, and it definitely would be very difficult to explain to him what a search query is and how dynamic directories work, so it is bewildering to imagine we would have to explain to him both. The problem is not just in explaining, it is in convincing. It is easy to put features and tell someone about them, but to (politely) force them into using them is something else. So, when designing the concept of Virtual Collections in GLScube, our ultimate goal was to wipe out traditional directories altogether, and in no way wanted the user to use both.

This is easy to do, and easier to say. But then what about the migration phase! Let us say that in 10 years time, everyone in the planet would be using GLScube, then till that day comes, we would have to maintain compatibility and interoperability between systems that use GLScube, and systems that don’t.

To make this more clear, let us say you created a Virtual Collection called “music” that, obviously, listed all of your music files. A friend came and gave you a CD containing some new song, you copied the song and put it in your GLScube Store. We ultimately don’t want you to specify any file path, and the song will, as you expect, always appear in the “music” Virtual Collection”. So where is the problem? The problem is in the answer to where that file was stored by GLScube. Since GLScube augments the traditional file system, it has to store it on the file system as a traditional file, which is really not a big deal, except that we would have to store it in an adhoc location. In other words, if the user wanted to retrieve this file from the underlying file system, say if he moved this drive to a non-GLScube system, then he would have a very hard time trying to find this file, which may even be named in a different name (or post-fixed with some value to distinguish duplicate names, or worse, placed in a some node in a hierarchy of directories, to avoid storage of too many files in one directory).

There are many such questions that have several answers. Like whether a user should be able to create a hierarchy of Virtual Collections, or that would put us where I started? Or, when a user decompresses a file, how would we deal with its parts that may depend on their hierarchy and each other?

This uncertainty in targets leads to uncertainty in design decisions, and also affects the scalability of the code.

Code. Code is another issue altogether. With the most critical point being consistency. Since GLScube runs in the user-space, and relies on inotify for receiving notifications on changes in the watched file systems, this kind of decoupling makes GLScube very susceptible to inconsistency due to unhandled events due to reasons like, most seriously, overflowing of the inotify buffer, and hence never knowing that some events even occurred!! I used PostgreSQL, which turned out not to be the best decision. Add to this some large number of stability issues, and it is suffice to say that the code base is not as good as it could be.

Now. Now we have two directions we can move to from here. We can continue working on the 0.1 prototype, or, we can start over! Starting over is not just about code-rewrite (which the current code base would really benefit from), but also a re-evaluation of all design decisions, mainly from the usability point of view.

I am most inclined to the second approach. The 0.1 prototype provided a very clear picture of what issues are in this type of project, and the priority of the challenges in it.

To learn from our own misfortune, it is clear that the essence of the problem is how to solve usability correctly. And since a few should never be allowed to decide on how others should use a software, I am thinking about an open-way to do a usability study; some rapid-prototyping method to put rough mockups for all the possible features and receive feedback from the open source community on what works better.

Note: For people who want instructions to compile the 0.1 prototype, we are preparing those and will put them together as soon as we can!

3 comments October 16th, 2006 amr.ramadan

A Rant about Filesystems of Today

If you grab the nearest Introduction to Computers book, and if that book were to mention the definition of a File System, you would find it claiming something like that a “File System is responsible for storing and organizing your files.” Sure, it stores the files and I am not going to discuss here whether they do that good enough or not, but what I am interested about is the orgnization claim. They do allow you organize, definitely, in those hierarchies, but is that good enough? In a day were each file has tons of information inside it, whether that file being text, an image, video or a complex vector animation. In a day where almost all popular file formats have rich metadata that describe the data they hold. In a day where it is no longer imagination to anticipate that sooner than later, you would be able to search through images that were analyzed and classified by complex algorithms. In this day, is it really good enough that all we have for organizing our files is one constraint: the name of the file and its location in some tree? I don’t think so.

It may be argued that I should not be ranting about File Systems, but rather, about the desktop environments. True, ResierFS never claimed to offer improved Usability, and hence, one may be inclined to think that it is all up to the desktop environments to offer a better interface for users to access their data.

However, researches in this field have not favoured one side of the camp. The Semantic File System by Gifford et al, one of the earliest approaches that attempted to improve the usability of file systems - which surprisingly, Gifford did not like in 1990 - merely augmented the underlying file system with a user-space daemon that indexes and extracts metadata from files, and an NFS server to provide users with a backward compatible interface. In his design, Gifford did not change how bits and bytes are written to whatever file system was used back then in 1990. Other projects went to the direction of extending the file systems with additional features that would allow better access and richer information to the stored data. In that context, SHORE actually stored typed objects on disk, rather than plain files. Others like BFS implemented metadata indexing within the context of the file system.

Hence comes the problem. First of all, in all I have read, I found no literature with a convincing argument of why an integrated solution would be better than an augmented one, or the other way around. Second, it is like these projects come and go. Many papers have been written, and prototypes produced in the past 15 years with the single goal of improving on the usability of the then-seen-limiting hierarchical file systems. And where are we now? Almost exactly as we were 15 years ago.

Almost, I say, almost. The one thing that recently got a push is what is now commonly known as Desktop Search. Most known are Google Desktop Search on Microsoft Windows, Spotlight on MacOSX and Beagle on GNU/Linux. However, I perceive desktop search as only more efficient programming; it is neither revolutionary, or even attempting to be.

Why is that you may ask? Let’s see. Why do you need to search? Because you cannot find something. But do you always search? No. You search when you cannot find something, or when you think that it may be difficult to find something. Searching, and according to many research findings, is more cognitively difficult than browsing. And although that perfect query with those two ANDs, an OR, and that sentence placed between double quotes could get you that difficult match in the first page of results, I would highly suspect your average user would be able to follow your suit.

Desktop Search tools are not “bad”. It is just that they are not enough. They sure would come handy if you are stuck and cannot find your file anywhere, but I do not see how they could become the primary mean of retrieving your files.

With the increase in the amount of storage available to users, average users will inevitably reach the point where they would not delete their files. It is already starting to be the case with many I know, those whose usage of computers is merely for word processing and e-mail. Then, what would be the case in a few years time when storage devices are in the units of Terabytes or perhaps more if some new technology broke to the streets. If you find it difficult now to find your way through hundreds or thousands of images, then how would it be 100 years from now when your grand-*-son has some million images of the family history.

And till when people would have to control the revisions of their files on their own. Again, this is a point that has been tackled by researchers but was never pushed for the masses: Version Controlled File Systems. Till when people would have to rename “CV.rtf” to “CV2.rtf” and with every few edits they would increment the name so that they would not lose a previous revision. Ah! And when they want to find their CV, they of course do not remember whether the right reversion was “CV6.rtf” or “CV7.rtf”.

Not only is the explosion in the amount of files is a huge issue, but also is the fact that users now maintain their files in separate storage entities, which could be in the same computer, in a neighboring computer, or in a computer in another continent than he is. Why would you always have to be aware of such dispersion? (If you are thinking NFS, wait, we will get to that later). Why do you have to maintain redundant copies, and why do you have to synchronize files between this and that computer, and between this computer and that portable device. Microsoft WinFS had some solutions for these issues, but they may not see the light anymore after they decided to dissect the 16-year old project into fragments.

What about Metadata? It is like they do not exist. Almost all of the most common file formats nowadays have Metadata that are left for the users to set. You know, this Author field in a PDF that is usually empty, and if not, probably has some cryptic irrelevant text. This is understandable of course, and users should not be blamed, because quite frankly my dear, Metadata means nothing. If they cannot organize based on it, or search with it, why would they spend the time to set it? And even more, why would they exercise the trouble in going through menus to edit them. Maybe, only a maybe, they would have set them if they were asked to in the Save dialogue of their application. But other than that, why spend the effort!?

While we are on the subject of Metadata, we need to discuss something relevant. Please, settle it down. Are you going to store Metadata in the file system or not!! I am simply not going to write an application that relies on storing Metadata on the disk if some file systems do not have such feature. The problem is not simple. Let us say that you made the perfect Storage/Organization solutiona and you wanted to allow users to add annotations to his documents. Now the GIF file format does not have room in its file format specification for such “Comments” data. One solution is to store this “extended” information on the file system. But if you were to store it in the file system, this means that moving the file to another system will move it without this extended information (you can make hacks, but nothing would be clean enough). It is a little heartbreaking that after we almost died in convincing the users to add semantics to his documents, we would then dumo away this information when he puts the file on his removable disk. This brings us back to the topic of augmented versus integrated semantic representation. We need to settle this issue. Will file systems go towards storage of bits only, or will they march towards additional territories like storing semantics.

Mind you, I know that most of the features I spoke of can be more or less glued to a Linux-based distribution. Someone could say, okay, get this library for searching, that library for version control, that on top of NFS to get an abstraction for dispersed storage devices, and… No. It is not that simple, because it never was a technical issue. If it were, we would have already been using all these features today.

I myself see it no big of a deal whether we implement a layer of semantics on top of the file system or inside the file system itself. Each solution has its advantages and disadvantages. What I care about is standardization. This is basically a usability issue. And it would go no one any good if searching was done by some software under KDE, by another under GNOME and by a third under Xfce. Maybe searching will not be that big of a deal, since as I said before, it never was the bigger issue. However, how would you like it if you have to organize your files in KDE based on Tags you assign and in GNOME based on the Metadata within the files!?

Again, if it was only a technical issue we would have already been semantic’ing around. The hierarchical organization scheme was molded around the traditional means of file cabinet organization. In other words, users can relate to it. Organizing your “files” into “folders” does not require a steep learning curve, at least compared to a semantic-based organization. Heck, I still find it difficult to explain to even the technically-inclined what semantic storage is!!

It is all about usability.

Gifford, in his paper titled Semantic File System, implemented what he called Virtual Folders. These folders were, contrary to the folders of the traditional file system, with no predefined content. Each Virtual Folder had associated with it a query, and when you would open that folder, that query would be executed and the results would then be put into that folder, at this instance of time. If you open that same folder one hour later and you had created a new file that matched that very same query, then when opening that Virtual Folder again you would find the new file in it, without you having to specifically put it in the Virtual Folder.

Virtual Folders could be now seen in Apple’s Spotlight, where they are called Smart Folders.

So, is this really what the user would all need!? Let us picture a little scenario. Let us say you frequently edit family videos, and thus, you created a Virtual Folder (or Smart Folder, or whatever the name is) called “Home Videos”, and you set its query to be “type:avi tag:toedit”. Now, since you really are not that geeky in video compression, a friend of yours once gave you the right settings to use when encoding your files, settings like the Bit Rate for example. You put his instructions into a file and you named it “encoding_settings.txt”. That file would of course not appear in the Virtual Folder; it is neither an AVI file nor it is assigned a tag “toedit”. The problem is, you want it to.

One easy solution to this problem is to just edit the query to something like “type:avi tag:toedit or name:encoding_settings.txt”. That would work, assuming there are no duplicates - that file your friend helped you with would appear right in your Virtual Folder, but is that the way it should be?

Well, there are other ways. One of them is to make Exclude and Include Lists for Virtual Folders. In an Include List, you would specify one file or more that must be included in the results. Whatever the result was, this file would appear in them if it was included. In other words, more fine-grained results at the cost of added complexity on the user.

Now that you have come to this point, you should have grasped what GLScube is about. If you did not, please allow me. GLScube is about providing users with means to organize and retrieve their files in the easiest possible way. It is about how to allow users to better store their data, how they can better access their data. GLScube is an attempt to provide an extensible standard way for organizing and retrieving your files.

You should realize that many are not yet in dead-need for semantic-based organization. But in few years, they certainly will. The constrains of performance of 15 years ago, which are less relevant now, will become easily ignorable in a few years. The technicalities are not rocket science too. The main issue is how to find a middle ground for usability, and then, how to convince users with the new idea of semantic storage and encourage them to move to it.

This rant is merely a short description of why GLScube was born. Unfortunately, with this driving force, we were unaware that the challenges would not be how to sort out technical issues, but how to solve usability issues. Tag-based organization is certainly fun, but is it the best there is?

I will next be writing more details on the technical issues, and the crazy choices we were faced with in terms of usability.

5 comments September 30th, 2006 amr.ramadan

What is, and Who is behind, GLScube

A little bit of history for those interested to know.

GLScube started as an idea between four people, namely, Ahmed Salama, Ahmed Samih, Amr Ramadan and Karim M. Yousef. All of us are (to be accurately, it is now “were”) students of Computer Engineering at the Alexandria University of Egypt. The idea was simple, we wanted to bring metadata, content and relational-based storage, organization and searching to GNU/Linux. We saw what Apple was doing with Spotlight, what Microsoft was doing with WinFS, what researchers have been doing for the past twenty years or so, and we were immediately hooked to the idea.

The reason for us being interested in a new idea for storage, organization and searching is that we felt that hierarchical organization, well, sucks! Thus, we went on to investigate researches that tackled this point. To our surprise, it wasn’t few. We read through many papers that dated as far as the late 80’s that presented alternatives to hierarchical organization.

“There are a number of usability problems to be overcome to the retrieval side that occur as a result of the inability to place documents into more than one category. Using a hierarchical system, the user is bound to a static organization scheme. Hence, he or she cannot view the information using a different scheme during retrieval than what was used when the information was initially organizaed.” (Quan et al in ‘User Interfaces for Supporting Multiple Categorization‘)

That was in summer, and after researching and digging in the past, we decided to continue work on this subject as our Graduation Project from the University. We worked on this under the supervision of Prof. M. A. Ismail, whose Doctoral is in Image Processing and Information Retrieval. From this point on things are pretty well documented in our Design Documentation, available at glscube.org. This chapter of the story ended with our presentation of the project and grabbing an “Excellent” degree for it (Excellent = As High As It Gets). Today, we are still working on it, and we plan on doing so.
So, that is some little info about who is behind GLScube, and a little bit of introduction of how it came to be. In later posts, I will dicuss what GLScube actually is, and where it is going to.

Add comment July 23rd, 2006 amr.ramadan

The State of GLScube

A Rant about Filesystems of Today

What is, and Who is behind, GLScube

Categories

Links

Feeds