Im(ai)gine

The current platforms (like replicate or huggingface) to fine-tune and share diffusion models are not designed to be marketplaces. They are designed to host the models and let you use them without rewarding the creators. However, I think that creators should be rewarded for their work. Fine tuning a model is not easy, especially when you want to have a good quality.

The Idea

The idea is very simple: allow users to fine-tune a model on their own data and share it on a marketplace using the SUI blockchain. Today one of the major issue when fine-tuning a model is the complexity of the process. It requires quite a lot of knowledge even on popular platforms like replicate or fal. The goal of Im(ai)gine is to make this process as simple as possible. With as little as 3$ you can fine-tune a model with your own data and share it on the marketplace.

Smart Contract

To make this possible, I used the power of the SUI blockchain and specifically the Kiosk primitive. Kiosk is a primitive provided by SUI to create a marketplace. It allows you to create a marketplace where you can buy and sell objects. However using Kiosk might be challenging and requires a lot of efforts.

Here is the model struct:

// model module
public struct Model has key, store {
    id: UID, // The id of the model
    owner: address, // The owner of the model
    weights_link: String, // The link to the weights of the model
    trigger_word: String, // The trigger word to use the model
    image_url: String, // The default image of the model
    is_published: bool, // Whether the model is published
    model_type: String, // The type of the model (either "people" or "style")
}

This model struct represents a fine-tuned model on the SUI blockchain. Through this NFT, you can buy and sell the model on the marketplace. And this allows the owner to earn royalties every time the model is used. Now let's talk about Kiosk. To begin with, we need to create a model_policy to restrict the transfer of the model. Indeed, we need to create an init function within the model module.

// model module
public struct MODEL has drop {}

#[allow(lint(share_owned))]
fun init(otw: MODEL, ctx: &mut TxContext) {
    let publisher = package::claim(otw, ctx);

    let (model_policy, model_policy_cap) = transfer_policy::new<Model>(&publisher, ctx);

    transfer::public_share_object(model_policy);
    transfer::public_transfer(model_policy_cap, ctx.sender());

    transfer::public_transfer(publisher, ctx.sender());
}

This function is used when we publish the smart contract, so it's executed only once. It creates a model_policy shared object to allow anyone to use this policy. Then it creates a model_policy_cap to transfer the policy to the owner of the smart contract.

Then when we want to buy a model, we will create a TransferRequest. This request must be confirmed with the model_policy object (basically here, we must confirm that the buyer fullfil the set of rules defined in the model_policy).

// main module
public fun buy_model(model: address, kiosk: &mut Kiosk, payment: Coin<SUI>, policy: &TransferPolicy<Model>): Model {
    let (model, request) = kiosk::purchase<Model>(kiosk, object::id_from_address(model), payment);

    confirm_request(policy, request);
    model
}

// model module
public fun confirm_request(policy: &TransferPolicy<Model>, request: TransferRequest<Model>) {
    transfer_policy::confirm_request<Model>(policy, request);
}

When the request is confirmed, the transaction can be finalized and the buyer will own the model.

Conclusion

I did this project during the Blockchain Student Association - EPFL x Sui Foundation hackathon on 12-13 October. I placed 4th with my project Imagine (the website may be unavailable) powered by Sui blockchain.