From Personal Information Management to Humane Interaction

While discussing file management in the paper on Laevo I presented today at the UIST conference, I conclude …

[...], in essence files are a remnant of the original desktop metaphor. Users are forced to mentally connect window representations to the files they represent. When restoring window configurations users are [unnecessarily] confronted with finding all the related files.

I reflect on this later in the discussion:

[...], raising interesting questions for further research on how window management can be redesigned to outgrow its original purpose. Further research on Laevo is therefore to increasingly move away from files, as their main intent of persisting information could be replaced by persisting window configurations [...]

This is in line with an old post of mine on window management, where I concluded:

Taking this to the extreme: assume closing a window would be the same as deleting a file. Would you actually ever have to know about the underlying file system again? Window management and file management could become one and the same thing.

Originally I titled the current post, “From File Management to Time Management”, since one of the conceptual challenges I like to confront myself with is to design for never having to reopen a file again. Rather, I want to support revisiting the full context (including the window representation of the file) which the original file was part of. As a desktop interface, Laevo uses a temporal representation allowing you to revisit any prior, or planned activity in time.

However, after a yet again inspiring talk by Bret Victor on humane interaction as the closing keynote of the UIST 2014 conference, I realized that just as file management is a remnant of the original desktop metaphor, so is window management. Windows are a side effect of the digital rectangles we’ve grown so accustomed to within our lives. Window representations are mere visual abstractions of richer concepts and ideas which could be expressed in entirely different ways using all of our senses, rather than being restricted to visual and symbolic notations. The reason why we stick to them is because they allow for dynamic (connected) behavior, which is where the tangible all-around-us world falls short. Following the same argument that we should be phasing out file management, so should we attempt to eliminate the need for window management. The more intermediate abstractions we can remove to interact with the concepts and ideas we actually want to address, the better.

Nonetheless, my underlying thesis remains. The temporal (and associated contextual) dimension is a very tangible, humane concept, we should continue to design for.

1 Comment

Laevo presentation @ UIST 2014 conference

A first paper on Laevo has been accepted to the ACM Symposium on User Interface Software and Technology (UIST) 2014 in Honolulu, Hawaii.

Laevo: A Temporal Desktop Interface for Integrated Knowledge Work

I will be presenting the 8th of October, including a live demo of the system. Looking forward to the conference! The publication includes a 30s teaser and a longer video showcasing the different supported interactions.

Leave a comment

Getting started with the Java SignalR SDK

Microsoft has just released a Java SignalR SDK which allows Java and Android clients to access ASP.NET SignalR back ends. The library is open source, and currently it is up to you to build the .jar packages yourself. For your convenience I’ve uploaded my built .jar files. Given that this library as of yet is still fairly undocumented, and I spent quite some time on getting it up and running, I figured I’d provide a short introduction tutorial here. I found the C# SignalR client documentation to be the most useful while figuring out the API as the classes and methods overall seem to correspond. The tests in the GitHub repository are another useful resource. I’ll mainly be focusing on the specific differences in the Java library, and refer you to the original documentation for more elaborate information.

We will create a simple Hub supporting bidirectional communication. A quick and easy way to get a C# back end up and running for the purpose of this tutorial is to self-host a SignalR server in a simple console or WPF application. The following C# code shows a simple hub with one method which can be called by the Java client, SendMessage.

public class MessageHub : Hub
{
	public static event Action<string, string> MessageReceived = delegate { };

	public void SendMessage( string name, string message )
	{
		MessageReceived( name, message );
	}
}

I added a static event handler which can be consumed by the console or WPF front end to display the received message. SendMessage is called from a different thread, so in case you want to update the UI, don’t forget to use a dispatcher!

MessageHub.MessageReceived  += ( name, message ) => _dispatcher.Invoke(
    () => { MessageBox.Content = String.Format( "{0} said {1}", name, message ); } );

Onwards to the client side code! To set up a Java project which can use the SignalR client API, add the ‘signalr-client-sdk.jar’ and ‘gson-2.2.2.jar’ to the project build path. For Android projects the references need to be added differently. Drag the two jars, and the additional ‘signalr-client-sdk-android.jar’ to the libs folder in Eclipse. By doing so they will automatically be added as Android libraries.

As mentioned before, the SignalR Java client follows the same structure as the C# client API, thus setting up a connection is quite similar. However, for Android applications an additional platform component needs to be loaded as shown below; also don’t forget to add internet permission to your manifest file, or you will receive a SocketException when trying to connect.

Platform.loadPlatformComponent( new AndroidPlatformComponent() );
// Change to the IP address and matching port of your SignalR server.
String host = "http://192.168.0.xxx:8080";
HubConnection connection = new HubConnection( host );
HubProxy hub = connection.createHubProxy( "MessageHub" );

This simply configures the connection. Establishing the actual connection is somewhat different than the C# documentation due to limitations of Java. To mimic language support for async in C#, the SignalRFuture class is introduced. Asynchronous operations return an instance of this class, without actually performing any real work yet. What follows is example code of how to start the connection synchronously by calling get() on the SignalRFuture. Don’t forget to cleanly stop() the connection when shutting down the application.

SignalRFuture<Void> awaitConnection = connection.start();
try {
	awaitConnection.get();
} catch (InterruptedException e) {
	// Handle ...
} catch (ExecutionException e) {
	// Handle ...
}

In case your server is up and running, you should now be ready to start listening to and submitting messages. Again, listening to events in C# is more straightforward since it supports lambdas as the C# client API documentation demonstrates.

stockTickerHubProxy.On<Stock>( "UpdateStockPrice", stock => Console.WriteLine(
    "Stock update for {0} new price {1}", stock.Symbol, stock.Price) );

Doing something similar in Java requires elaborate inline anonymous classes, or the creation of a handler class per event you want to listen to. Suppose the server would send a message “context.Clients.All.UpdateStatus( “Online” );”, handling this the ‘C# way’ would like as follows:

hub.on( "UpdateStatus",
    new SubscriptionHandler<String>() {
        @Override
        public void run( String status ) {
           // Since we are updating the UI,
           // we need to use a handler of the UI thread.
           final String fStatus = status;
           handler.post( new Runnable() {
               @Override
               public void run() {
                   statusField.setText( fStatus );
               }
           } );
        }
    }
, String.class );

Therefore, Microsoft has seemingly added a method not available in the C# API to the Java client. Calling hub.subscribe( listener ) where listener is an object implementing corresponding methods for every available incoming event is a much more straightforward way to listen to messages. Under the covers this uses reflection to hook everything up correctly. These methods need to be public!

hub.subscribe( this );
...
public void UpdateStatus( String status )
{
	final String fStatus = status;
	handler.post(new Runnable(){
		@Override
		public void run() {
			statusField.setText( fStatus );
		}});
}

More complex types work as well, as long as it is supported by JSON. I don’t know the specifics, but this will be dependent on the SignalR server and the gson library used by the SignalR Java client. The following code prepares the client to call a new method on the server which you can add to the MessageHub: “public void SendCustomType( CustomType object ) { … }“.

// A simple C# class which can be sent over SignalR.
public class CustomType
{
	public string Name;
	public int Id;
}
// The same type as defined in Java.
public class CustomType
{
	public String Name;
	public int Id;
}

Finally, calling the earlier SendMessage() and the newly added SendCustomType() can be done as follows. As you can see, remote method invocation again returns a SignalRFuture, and a subsequent get() is needed.

try {
    hub.invoke( "SendMessage", "Client", "Hello world!" ).get();
    hub.invoke( "SendCustomType",
        new CustomType() {{ Name = "Universe"; Id = 42; }} ).get();
} catch (InterruptedException e) {
	// Handle ...
} catch (ExecutionException e) {
	// Handle ...
}

Similarly, you can simply pass custom objects from the server to the client by adding the custom type as a method parameter: “public void SomeMethod( CustomType bleh ) { … }

12 Comments

Generic Attributes in C#

As awesome as C# is, once you want to do some more advanced stuff with attributes, you quickly run into several limitations. There is even a Microsoft Connect entry on this issue.

  • Attributes can’t be generic, since a generic type cannot derive from ‘Attribute’.
  • An attribute argument must be a constant expression, typeof expression or array creation expression of an attribute parameter type.

Using TypeDescriptor and a whole bunch of custom code these limitations can be overcome. In this post I will describe an easier straightforward way by which to create something similar to generic attributes, ignoring the second issue for now. In future posts I will discuss how and where these attributes can be used for something I previously called attribute metabehavior.

The idea is quite straightforward; although attributes can’t be generic, nothing prevents you from adding generic instances to them. These instances can later be extracted at run time using reflection. The trick of course is initializing the generic instance. Once you know the Activator class can create instances by knowing their Type definition, the solution becomes self-evident. By simply passing the generic type as an argument to the attribute, instance creation can be delegated to Activator. Optional constructor arguments can also be passed as attribute arguments.

public class BehaviorAttribute : Attribute
{
	/// <summary>
	///   The dynamically created instance of the type passed to the constructor.
	/// </summary>
	public object DynamicInstance { get; private set; }

	/// <summary>
	///   Create a new attribute and initialize a certain type.
	/// </summary>
	/// <param name = "dynamicType">The type to initialize.</param>
	/// <param name = "constructorArguments">
	///   The arguments to pass to the constructor of the type.
	/// </param>
	public BehaviorAttribute(
		Type dynamicType,
		params object[] constructorArguments )
	{
		DynamicInstance =
			Activator.CreateInstance( dynamicType, constructorArguments );
	}
}

This attribute can then be be applied as follows:

class Answer<T>
{
	public T Value;

	public Answer( T value )
	{
		Value = value;
	}
}

[BehaviorAttribute( typeof( Answer<int> ), 42 )]
class TheWorld {}

Using ordinary reflection, the instance can be extracted.

Type type = typeof( TheWorld );
var behavior = (BehaviorAttribute)type
    .GetCustomAttributes( typeof( BehaviorAttribute ), false ).First());
var genericInstance = (Answer<int>)behavior.DynamicInstance;
int answer = genericInstance.Value;

What follows is up to you, how will you use this? I’ll describe a few advanced use cases in subsequent posts.

5 Comments

Start of Laevo user studies

timeline_smaller

As part of my PhD I created Laevo, an alternate way by which work can be organized under Windows 7/8, and now it needs to be evaluated by several participants during a 2 week study. Laevo augments your current windows environment with a couple of extra options to organize your daily activities. The goal is you attempt to use these features during a full 2 week period, while continuing doing the activities you ordinarily do. How much you want to use the system is entirely up to you, but ideally you have it running during the full 2 weeks. You can either exit Leavo at the end of each day, or use Window’s sleep and hibernate functionality.

You would help me out greatly by installing Laevo, and trying it out. You can start the 2 week period of using it either on Monday the 5th, 12th or 19th of August. At the end of each day please give some feedback on your experiences with the system that day by shortly answering a set of questions. This shouldn’t take longer than 5 minutes/day. You can compile the feedback in one document and send it to me at the end of the 2 weeks (sjeu AT itu.dk).

  1. Why was or wasn’t Laevo useful for you today? At a minimum state one positive and one negative points, but open feedback is encouraged.
  2. What activities have you done today that weren’t represented in Laevo at some point?
  3. What was/were your main activities today? In case they were represented in Laevo, where did they originate from (self-initiated, to-do item, email to-do, other)?
  4. Have you scheduled any activities today? Did you also plan them on the time line? Why (not)?
  5. Did you use Leavo’s to-do list today? Why (not)? How?
  6. Did you use Laevo’s Activity Context library today to store or retrieve files?
  7. Were there occasions where you considered creating an activity or to-do item but eventually decided not to? If so, why?
  8. Please have a look at your time line. Does the overview of today reflect the actual activities you did today? Why (not)?

You can contact me for any information on sjeu AT itu.dk, but I will be out of office until the 11th of August.

When the application crashes there should be a “log.txt” file available in “C:\Users\<username>\Documents\Laevo”. Please email this to me. In case you continue encountering problems which severely hinder you from your work, please report them so I can try sending you a new version of Laevo to resolve the issues.

Please forward this to any participants which might be interested. Thank you for helping me out!

Leave a comment

Call for Laevo user study participants

Custom

With great pleasure I can finally announce the first public installer of Laevo, a project I started out working on as part of my master’s thesis, and am now continuing working on as part of my PhD. In short, it allows you to organize your work in new ways which Windows traditionally doesn’t offer you. In case you spend most time of the day working on your PC you might be interested in trying it out. The project has come a long way and is finally ready for the greater public, hence I am looking forward to your input on what you think about the system. You would help me out greatly by installing Laevo, trying it out for one to two weeks, and giving feedback on it afterwards. Please send me an email (sjeu at itu.dk) or simply contact me in case you are interested in participating.

timeline_smaller

You can download the latest version (v0.1.3) here. This is a more recent version than the one I linked to on twitter earlier this week. In case you already installed the previous version, simply uninstall the old version and install the new one. Your data and settings will be saved. To get you started I strongly advise you to quickly read through the manual so you know about the functionality offered.

4 Comments

Chrome as default PDF reader

Let me start by venting a bit and expressing my deep displeasure with the latest Foxit reader 6 release. I originally installed Foxit as a lightweight replacement for Adobe reader, but the latest release lacks the main advantage earlier versions had – quick instantaneous PDF access. A colleague of mine recommended using Chrome’s built in PDF reader as a default reader instead since I’m already using Chrome either way. As you might have experienced when opening PDFs from the web, it’s lightning fast in Chrome. This post isn’t just about how to set up Google Chrome as your default PDF reader, which is rather straightforward; I’ll gladly refer you to How-To Geek for help with that. However, once you have done so you might notice some undesirable behavior which differs from using a dedicated PDF reader.

PDF file with gray borders on the sides.

  • When you already have a browser window open, the PDF opens as a new tab in that browser. This might actually be considered a feature by some, but as I blogged before, I rather move away from application-specific tab management entirely.
  • The chrome ‘PDF window’ doesn’t retain its own size when opening PDFs, but shares it with regular browser usage. PDFs are generally not that wide, and maximizing the window results in gray borders on the sides.

After some hacking around I found a solution around this. By placing a custom executable (download here) in the same folder as chrome.exe and using this executable as the default application to open PDF files, both before mentioned issues are solved. Double clicking a PDF file results in a new chrome window, having the same size as the last PDF file you opened.

The chrome executable by default is located in: C:\Users\<user>\AppData\Local\Google\Chrome\Application\

How it works

The executable simply calls the original chrome.exe, but additionally adds two command line arguments.

  • –user-data-dir: Specifies the user data directory, which is where the browser will look for all of its state.
  • –new-window: Launches URL in new browser window.

The user data directory besides other settings also seems to contain the last set window size. Passing any non existing path here results in a new directory being created for it under “C:\Users\<user>\AppData\Local\Google\Chrome\Application\<version_number>”, containing the settings which will be reused the next time the executable is called. In case you want to adjust the behavior (e.g. disable opening the PDFs in a new window) what follows is the C# source code for the executable. Originally I tried creating a shortcut and setting that as the default application to open PDFs, but the executable to which the shortcut points ends up being used instead, hence losing the command line arguments.

static void Main( string[] args )
{
	string chromeDir =
		new FileInfo( Assembly.GetEntryAssembly().Location ).DirectoryName;
	string chrome = Path.Combine( chromeDir, "chrome.exe" );
	string arguments =
		"\"" + args[ 0 ] + "\"" + " --user-data-dir=pdf_dir --new-window";

	var proc = new Process
	{
		StartInfo =
		{
			FileName = chrome,
			Arguments = arguments,
			WorkingDirectory = chromeDir
		}
	};

	proc.Start();
}

Leave a comment

Follow

Get every new post delivered to your Inbox.