custom

If you have been reading bindpose for a while and have seen my marking menu posts you probably know that I am very keen on getting my workflow as optimized as possible. I am a big fan of handy shelfs, marking menus, hotkeys, custom widgets, etc. The way I see it, the closer and easier our tools are to access the quicker we can push the rig through. That is why today we are having a look at using PySide to install a global hotkey in Maya (one that would work in all windows and panels) in a way where we do not break any of the existing functionality of that hotkey (hopefully).

If you have not used PySide before, do not worry, our interaction with it will be brief and pretty straightforward. I myself am very new to it. That being said, I think it is a great library to learn and much nicer and more flexible than the native maya UI toolset.

Disclaimer: The way I do this is very hacky and dirty and I am sure there must be a nicer way of doing this, so if you have suggestions please do let me know, so I can add it both to my workflow and to this post.

What we want to achieve

So, essentially, all I want to do here is install a global hotkey (PySide calls them shortcuts) on the CTRL + H combination that would work in all Maya’s windows and panels as you would expect it to – Hide selected, but if we are inside the Script editor it would clear the history.

Some of you might think that we can easily do this without PySide, just using maya’s hotkeys, but the tricky bit comes in from the fact that maya’s hotkeys are not functioning when your last click was inside the Script editor’s text field or history field. That means, that only if you click somewhere on the frames of the Script editor would that hotkey get triggered, which obviously is not nice at all.

Achieving it

So, let us have a look at the full code first and then we will break it apart.

from functools import partial
from maya import OpenMayaUI as omui, cmds as mc

try:
    from PySide2.QtCore import *
    from PySide2.QtGui import *
    from PySide2.QtWidgets import *
    from shiboken2 import wrapInstance
except ImportError:
    from PySide.QtCore import *
    from PySide.QtGui import *
    from shiboken import wrapInstance


def _getMainMayaWindow():
    mayaMainWindowPtr = omui.MQtUtil.mainWindow()
    mayaMainWindow = wrapInstance(long(mayaMainWindowPtr), QWidget)
    return mayaMainWindow


def shortcutActivated(shortcut):
    if "scriptEditor" in mc.getPanel(wf=1):
        mc.scriptEditorInfo(clearHistory=1)
    else:
        shortcut.setEnabled(0)
        e = QKeyEvent(QEvent.KeyPress, Qt.Key_H, Qt.CTRL)
        QCoreApplication.postEvent(_getMainMayaWindow(), e)
        mc.evalDeferred(partial(shortcut.setEnabled, 1))


def initShortcut():
    shortcut = QShortcut(QKeySequence(Qt.CTRL + Qt.Key_H), _getMainMayaWindow())
    shortcut.setContext(Qt.ApplicationShortcut)
    shortcut.activated.connect(partial(shortcutActivated, shortcut))

initShortcut()

Okay, let us go through it bit by bit.

Imports

We start with a simple import of partial which is used to create a callable reference to a function including arguments. Then from maya we the usual cmds, but also OpenMayaUI which we use to get a PySide reference to maya’s window.

Then the PySide import might look a bit confusing with that try and except block, but the only reason it is there is because between maya 2016 and maya 2017 they switched PySide versions, and the imports had to change as well. So, what we do is we try to import from PySide2 (Maya 2017) and if it cannot be found we do the imports from PySide (Maya 2016).

Getting Maya’s main window

Even though, Maya’s UI is built entirely by Qt (PySide is a wrapper around Qt), the native elements are not usable with PySide functions. In order to be able to interact with these native bits we need to find a PySide reference to them. In the example for hotkeys we need only the main window, but depending on what you are trying to do you might have to iterate through children in order to find the UI element you are looking for. Therefore this _getMainMayaWindow function has become a boilerplate code and I always copy and paste it together with the imports.

The way it works is, using Maya’s API we get a pointer to the memory address where Maya’s main window is stored in memory. That’s the omui.MQtUtil.mainWindow() function. Then what we do is, using that pointer and the wrapInstance function we create a PySide QWidget instance of our window. That means that we can run any QWidget functions on Maya’s main window. In our hotkey example, though, we only need it to bind the hotkey to it.

The logic of the hotkey

The shortcutActivated function is the one that is going to get called every time we press the hotkey. It takes a QShortcut object as an argument, but we will not worry about it just yet. All we need to know is that this object is what calls our shortcutActivated function.

It is worth mentioning that this function is going to get called without giving Maya a chance to handle the event itself. So, that means that if we have nothing inside this function, pressing CTRL + H will do nothing. Therefore, we need to make sure we implement whatever functionality we want inside of this function.

So, having a look at the if statement, you can see that we are just checking if the current panel with focus – mc.getPanel(wf=1) – is the Script editor. That will return True if we have last clicked either on the frames of the Script editor windows or anywhere inside of it.

Then, obviously, if that is the case we just clear the Script editor history.

If it returns False, though, it means that we are outside of the Script editor so we need to let Maya handle the key combination as there might be something bound to it (In the case of CTRL+H we have the hiding functionality which we want to maintain). So, let us pass it to Maya then.

As I said earlier, Maya does not get a chance to handle this hotkey at all, it is entirely handled by PySide’s shortcut. So in order to pass it back to Maya, what we do is we disable our shortcut and we simulate the key combination again, so Maya can do it’s thing. Once that is done, we re-enable our shortcut so it is ready for next time we press the key combination. That is what the following snippet does.

shortcut.setEnabled(0)
e = QKeyEvent(QEvent.KeyPress, Qt.Key_H, Qt.CTRL)
QCoreApplication.postEvent(_getMainMayaWindow(), e)
mc.evalDeferred(partial(shortcut.setEnabled, 1))

Notice we are using evalDeferred as we are updating a shortcut from within itself.

Binding the function to the hotkey

Now that we have all the functionality ready, we need to bind it all to the key combination of our choice – CTRL + H in our example. So, we create a new QShortcut instance, which receives a QKeySequence and parent QWidget as arguments. Essentially, we are saying we want this key combination to exist as a shortcut in this widget. The widget we are using is the main maya window we talked about earlier.

Then, we use the setContext method of the shortcut to extend it’s functionality across the whole application, using Qt.ApplicationShortcut as an argument. Now the shortcut is activated whenever we press the key combination while we have our focus in any of the maya windows.

Lastly, we just need to specify what we want to happen when the user has activated the shortcut. That is where we use the activated signal of the shortcut (more info on signals and slots) and we connect it to our own shortcutActivated function. Notice that we are using partial to create a callable version of our function with the shortcut itself passed in as an argument.

And that’s it!

Conclusion

Hotkeys, marking menus, shelves, custom widgets and everything else of the sort is always a great way to boost your workflow and be a bit more efficient. Spending some time to build them for yourself in a way where you can easily reproduce them in the next version of Maya or on your next machine is going to pay off in the long run.

I hope this post has shown you how you can override maya’s default hotkeys in some cases where it would be useful, while still maintaining the default functionality in the rest of the UI.

If you know of a nicer way of doing this, please do share it!

So, painting skin weights. It is a major part of our rigging lives and sadly one of the few bits, together with joint positioning, that we cannot yet automate, though in the long run machine learning will probably get us 99% there. Untill then though, I thought I would share some of my tips for painting skin weights with maya’s native tools, since whenever I would learn one of these I felt stupid for not finding it out earlier as, more often than not, it was just so simple.

I am sure a lot of you are familiar with these, but even if you learn just a single new idea about them today, it might boost your workflow quite a bit. Additionally, I know that a lot of you are probably using ngSkinTools and literally everyone I know who works with it says they cannot imagine going back. So I am sure that some of the things I am going to mention are probably already taken care of ngSkinTools, but if you, like me, have not had the chance to adopt it yet, you might find these helpful.

I am going to list these in no particular order, but here is a table of contents.

Contents

  1. Simplifying geometries with thickness and copying the weights
  2. Using simple proxy geometry to achieve very smooth weights interpolation quickly
  3. Duplicate the geometry to get maya default bind on different parts
  4. Copy and paste vertex weights
  5. Use Post as normalization method when smoothing
  6. Move skinned joints tool
  7. Reveal selected joint in the influence list
  8. Some handy hotkeys
  9. Average weights
  10. Copy and paste multiple vertex weights with search and replace
  11. Print weights

So with that out of the way, let us get on with it.

Simplifying geometries with thickness and copying the weights

This one comes in very handy when we are dealing with complex double-sided geometries (ones that have thickness). The issue with them is that when you are painting one side, the other one is left unaffected, so as soon as an influence object transforms the two sides intersect like crazy. That is often the case with clothes and wearables in general.

The really easy way to get around this is to
1. Make a copy of the geometry
2. Remove the thickness from it (when having a good topology it is as simple as selecting the row of faces which creates the thickness and deleting it together with one side of the geo)
3. Paint the weights on that one
4. Copy the weights back to the original geometry

Painting skin weights tips - Using a one sided proxy geometry when working with thickness

Now, a really cool thing that I had not thought of untill recently is that even if I have started painting some weights on the double sided geometry to begin with I can also maintain them, by copying the weights from the original one to the simplified one before painting it, so I have a working base.

That means, that if I have managed to paint some weights on a double sided geometry that kind of work, but the two sides are not behaving 1 to 1, I can create a simplified geo, copy the weights from the original one to the simplified and then copy them back to get the 1 to 1 behaviour I am looking for.

Using simple proxy geometry to achieve very smooth weights interpolation quickly

This one is very similar to the first one, but I use it all the time and not only on double-sided geometries.

Very often there are geometries that have some sort of a detail modeled in them that make it hard for weight painting smooth weights around it.

Consider the following example. Let us suppose that we need this geometry to be able to stretch smoothly when using the .translateX of the end joint.

Tips for painting skin weights in maya - Using a simple geometry to copy weights to models which are hard to smooth weights for.

Doesn’t look great with default skinning, but also if I try to block in some weights and smooth them, it is likely that maya won’t be able to interpolate them nicely. To go around it, I’d create a simple plane with no subdivisions so I can have a very nice smooth interpolation from one edge to the other.

Tips for painting skin weights - Using a simple plane without subdivisions to achieve a smooth weights interpolation for copying to complex geometries.

Copying this back to the initial geometry gives us this.

Tips for painting skin weights - Smooth skinned complex geometry using weights from a simple plane.

Very handy for mechanical bits that have some detail in them and also need to be stretched (happens very often in cartoon animation).

Duplicate the geometry to get maya default bind on different parts

So, very often I have to paint the weights on a part of a geometry to a bunch of new joints while I still need to maintain the existing weights on the rest of it. More often than not, I would be satisfied with maya’s default weights after a bind, but obviously if I do that it will obliterate my existing weights.

What I do in such cases is make a copy of the geometry and smooth bind it to only the new joints. Then I select the vertices on the original geometry that comprise the part I want the new influences in and I use the Copy skin weights from the duplicated one to the selected vertices. If the part is actually separated from the rest of the geometry that should do it, but if it s a more of an organic shape, there is going to be some blending of the new weights with the ones surrounding them.

I could imagine, though, that having the ability to have layers and masks on your skin weights would make this one trivial.

Copy and paste vertex weights

I am guilty of writing my own version of this tool just out of the ignorance of not knowing that this exists. Basically what you can do is select a vertex, use the Copy vertex weights then select another one (or more than one) and use the Paste vertex weights command to paste them. Works cross-geometries as well.

A cool thing about the tool that I wrote to do this is I added a search and replace feature that would apply the weights to the renamed joints. For example if I am copying a vert from the left arm and I want to paste it on the right I would add “L_” to “R_” to my replacement flags.

Use Post as normalization method when smoothing

So, I have met both people who love and who hate post. I think the main reason people dislike it is because they don’t feel comfortable with their weights being able to go above 1.0, but I have to say that sometimes it is very handy. Especially for smoothing. Everyone knows how unpredictable maya’s interactive smoothing is, and that’s understandable since in a lot of cases it is not immediately obvious where should the remaining weights go to.

Smoothing on post is 100% predictable which I think is the big benefit. The way it works is that it smooths out a joint’s influence by itself, without touching any of the other weights. That means that the weights are not normalized to 1.0, but instead of verts shooting off in oblivion post normalizes them for our preview. That is also why it is not recommended to leave skinClusters on Post as the weights are going to be normalized on deformation time which would be slower.

So more often than not my workflow for painting weights would be to block in some harsh rigid weights, then switch to Post and go through the influences one by one flooding them with the Smooth paint operation once or twice.

Move skinned joints tool

I am not sure which version of maya did this tool come in, but I learned of it very recently. Essentially you can select a piece of geo (or a joint) and run the Move skinned joints tool, then you can transform the joint however you like or you can also change the inputs going into it without affecting the geometry, though, you’d have to be careful to not change the tool or the selection as that would go out of the Move skinned joints tool. Ideally any other changes than just moving/rotating them about should be ready to be ran in the script editor.

I would not recommend using this for anything else than just testing out different pivot points. Doing it for actual positioning in the final skinCluster feels dirty to me.

Reveal selected joint in the Paint skin weights tool influence list

Only recently I found out what this button does.

Paint skin weights tool - reveal selected joint in influence list

It scrolls the list of influences to reveal the joint that we have selected, which is absolutely brilliant! Previously, I hated how when I need to get out of the Paint skin weights tool and then get back inside of it, the treeView is always scrolled to the top of the list. Considering that the last selection is maintained, pressing that button will always get you back to where you left off. Even better, echoing all commands gives us the following line of MEL that we can bind to a hotkey.

artSkinRevealSelected artAttrSkinPaintCtx;

Some handy hotkeys

I have learned about some of these way too late, which is a shame, but since then I’ve been using them constantly and the speed increase is immense. I hate navigating my mouse to the tool options just to change a setting or value.

  • CTRL + ALT + C – Copy vertex weights
  • CTRL + ALT + V – Paste vertex weights
  • N + LMB (drag) – Adjust the value you are painting with
  • U + LMB – Marking menu to change the current paint operation (Replace, Add, etc.)
  • ALT + F – Flood surfaces with current values

For more of these head on to the Hotkey editor, then in the Edit hotkeys for combobox go for Other items and open up the Artisan dropdown.

From here on I have added some of the functionalities that I have written for myself, but sadly the code is very messy to be shared. Luckily, it is not hard at all to write your own (and it will probably be much better than mine), but if you are interested, do let me know and I can clean it up and share it at some point.

Average weights

This one I use a lot. What it does is, it goes through a selection of verts and calculates the average weights for all influences and then goes through the selection once more and applies that average calculated weight. Essentially, what this results in is a rigidly transforming collection of verts. Stupidly simple, but very useful when rigging mechanical bits, which should not deform. Also I have used it in the past on different types of tubings and ropes where there are bits (rings, leafs, etc.) that need to follow the main deformation but not deform themselves.

Copy and paste multiple vertex weights with search and replace

In addition to the above mentioned copy and paste vertex weights, I have written a simple function that copies a bunch of vertex weights and then pastes them to the same vertex IDs on a new mesh. It is not very often that we have geometries that are copies of each other, but if we do this tool saves me a lot of time, because I can then just skin one of them, copy the weights for all verts and then paste them to the other geometry using the Search and Replace to adjust for the new influences.

Comes in particularly handy for radially positioned geometries where mirroring will not help us a lot.

Print weights

Quite often I’d like to debug why something is deforming incorrectly, but scrolling through the different influences can get tedious especially if you have a lot of them. So I wrote a small function that finds the weights on the selected vertex and prints them to me.

This is the kind of output I get from it.

#####################################
joint2 : 0.635011218122
joint1 : 0.364988781878
#####################################

As I said, there is a lot of room for improvement. It works only on a single vert at the moment, but I could imagine it being really cool to see multiple ones in a printed table similar to what you would get in the component editor.

What would be even cooler would to use PySide to print them next to your mouse pointer.

Conclusion

Considering that we spend such a big chunk of our time on painting weights we should do our best to be as efficient and effective as possible. That is the reason I wanted to share these, as they have helped me improve my workflow immensely and I hope you would find some value in them as well.

So, recently I stumbled upon a djx blog blost about custom hotkeys and marking menus in different editors in maya. I had been thinking about having a custom hotkey marking menu, but was never really sure how to approach this, so after reading that post I thought I’d give it a go and share my process.

tl;dr: We can create a runtime command which builds our marking menu and have a hotkey to call that command. Thus, giving us the option to invoke custom marking menus with our own custom hotkeys, such as Shift+W or T for example, and a mouse click.

Disclaimer: I have been having a super annoying issue with this setup, where the “release” command does not always get called, so the marking menu is not always deleted. What this means is that if you are using a modifier like Shift, Control or Alt sometimes your marking menu will still be bound to it after it has been closed. Therefore, if you are using something like Shift+H+LMB, just pressing Shift+LMB will open it up, so you lose the usual add to selection functionality. Sure, to fix it you just have to press and release your hotkey again, but it definitely gets on your nerve after a while.

If anyone has a solution, please let me know.

I have written about building custom marking menus in Maya previously, so feel free to have a look as I will try to not repeat myself here. There I also talked about why I prefer to script my marking menus, instead of using the Marking menu editor, and that’s valid here as well.

So, let us have a look then.

RunTimeCommands

The first thing we need to do is define a runTimeCommand, so we can run it with a hotkey. That is what happens if you do it through the Marking menu editor and set Use marking menu in to Hotkey Editor, as well.

There a couple of ways we can do that.

Hotkey Editor

On the right hand side of the hotkey editor there is a tab called Runtime Command Editor. If you go on that one you can create and edit runTime commands.

Scripting it in Python

If you have multiple marking menus that you want to crate, the hotkey editor might seem as a bit of a slow solution. Additionally, if changes need to be made I always find it more intuitive to look at code in my favourite text editor (which is sublime by the way).

To create a runTime command we run the runTimeCommand function which for some reason does not appear in the Python docs, but I’ have been using maya.cmds.runTimeCommand successfully.

All we need to provide is a name for the command, some annotation – ann, a string with some code – c and a language – cl.

Here is an example

mc.runTimeCommand("exampleRunTimeCommand", ann="Example runTime command", c=commandString, cl="python")

Something we need to keep in mind when working with runTime commands is that we cannot pass external functions to them. We can import modules and use them once inside, but I cannot pass a reference to an actual function to the c flag, as I would do to menuItems for example. That means that we need to pass our code as a string.

Press and release

Now, that we know how to create the runTimeCommands let us see what we need these commands for.

As I mentioned, they are needed so we can access them by a hotkey. What that hotkey should do is initialize our marking menu, but once we release the key it should get rid of it, so it does not interfere with other functions. Therefore we need two of them – Press and Release.

Let us say we are building a custom hotkey marking menu for weight painting. In that case we will have something similar to the following.

  • mmWeightPainting_Press runTimeCommand – to initialize our marking menu
  • mmWeightPainting_Release runTimeCommand – to delete our marking menu

The way we bind the release command to the release of a hotkey is by pressing the small arrow to the side of the hotkey field.

Custom hotkey marking menu - release command hotkey

The Press command

import maya.cmds as mc # Optional if it is already imported

name = "mmWeightPainting"
if mc.popupMenu(name, ex=1):
    mc.deleteUI(name)

popup = mc.popupMenu(name, b=1, sh=1, alt=0, ctl=0, aob=1, p="viewPanes", mm=1)

import mmWeightPainting
reload(mmWeightPainting)

So, essentially what we do is every time we press our hotkey, we delete our old marking menu and rebuild it. We do this, because we want to make sure that our latest changes are applied.

Now, the lower part of the command is where it gets cool, I think. We can store our whole marking menu build – all menuItems – inside a file somewhere in our MAYA_SCRIPT_PATH and then just import it from the runTimeCommand as in this piece of code. What this gives us, is again, the ability to really easily update stuff (not that it is a big deal with marking menus once you set them up). Additionally, I quite like the modularity, as it means we can have very simple runTimeCommands not cluttered with the actual marking menu build. This is the way that creating through the Marking menu editor works as well, but obviously it loads a MEL file instead.

So, literally that mmWeightPainting file is as simple as creating all our marking menu items.

import maya.cmds as mc

mc.menuItem(l="first item")
mc.menuItem(l="second item")
mc.menuItem(l="North radial position", rp="N")
...

And that takes care of building our marking menu when we press our hotkey + the specified modifiers and mouse button. What, we do not yet have is deleting it on release, so it does not interfere with the other functionality tied to modifier + click combo. That is where the mmWeightPainting_Release runTimeCommand comes in.

The Release command

## mmWeightPainting_Release runTimeCommand
name = "mmWeightPainting"

if mc.popupMenu(name, ex=1):
    mc.deleteUI(name)

Yep, it is a really simple one. We just delete the marking menu, so it does not interfere with anything else. Essentially, the idea is we have it available only while the hotkey is pressed.

Hotkey

All that is left to be done is to assign a hotkey to the commands. There are a couple of things to have in mind.

If you are using modifiers for the popupMenu command – sh, ctl or alt – then the same modifiers need to be present in your hotkey as otherwise, even though the runTimeCommand will run successfully, the popupMenu will not be triggered.

In the above example

mc.popupMenu(name, b=1, sh=1, alt=0, ctl=0, aob=1, p="viewPanes", mm=1)

we have specified the sh modifier. Therefore, the Shift key needs to be present in our hotkey.

Also, obviously be careful which hotkeys you overwrite, so you do not end up causing yourself more harm than good.

Conclusion

That’s it, it really is quite simple, but it helps a lot once you get used to your menus. Honestly, trying to do stuff without them feels so tedious afterwards.

I am not sure what it is, but there is something incredibly appealing in optimizing our workflows. I think a lot of it comes from the frustration of repeating the same actions over and over again. When you find a way to optimize that, it feels great. One of the easiest way to improve our rigging workflow is to script a custom marking menu with python. Another one, I have already written about is creating a custom shelf.

tl;dr: I will walk you through scripting your own custom marking menu with python, which is going to be easily shareable, extendable and maintainable. The code can be found here.

Here is how my main marking menu looks.

Custom marking menu in maya with python

I have found it is an immense help to have the commands I use most oftenly either in my marking menu or my shelf. It just saves so much time!

Okay, how do we go about creating one of these?

Well, we have two options – either build it with Maya’s native marking menu editor or script it with python.

The reasons I prefer scripting my marking menus in python are a few.
– The native editor does not give us all the available options for a marking menu, such as submenus.
– Updating from the editor is a pain in the butt.
– The editor does not scale nicely, if you want or need to support multiple marking menus.
– Doing it through the editor is boring.

Obviously, python fixes all these issues for us. Additionally, it is easy to share it with co-workers and keep it in a version control system. Okay, so you are sold now. Let us have a look at how to do it then.

The code that I will be going through is on this gist, but I will go through all of it, if you would rather write it yourself.

Custom marking menu with Python

Initialization

import maya.cmds as mc

MENU_NAME = "markingMenu"

We start very simple with the import of maya.cmds and giving a name to our custom marking menu. Now, the name is not very important because we do not ever see it, but maya does. So, in order for us to be able to update our custom marking menu, we need to be able to access it, and that is why we are giving it a name.

Then we have our markingMenu class. The main reason I went for a class is because we can encapsulate everything we need into it quite logically. I know that some people would prefer to have a function instead, which is absolutely fine, it is really a personal preference. Let’s have a look at the constructor.

Constructor

def __init__(self):

self._removeOld()
self._build()

As you can see, our constructor is very simple. We delete the old version of our custom marking menu if it exists and then we build are our new one in place.

Remove old marking menu

def _removeOld(self):
    if mc.popupMenu(MENU_NAME, ex=1):
    mc.deleteUI(MENU_NAME)

As I said, the reason we need to give a name to our marking menu is to be able to modify it after it is created. In our case, we are not really modifying it, but deleting it instead. We do this, so we have a clean slate for building our new marking menu.

The reason I have added this deleting and then building again functionality is just so we can painlessly make changes to our custom marking menu. For example, if I want to add a new button or change a label, I do not want to have to restart maya or do anything other than just running my code again. Or even easier, just importing my code again. That is why I said earlier, that this setup for a custom marking menu is very easily extendable and maintainable. Additionally, we can add a button to our marking menu which rebuilds it, so we only have to make our changes to the file, save it and then rebuild from within. We will have a look at that in the end of the post.

Building our custom marking menu

Now that we have had a look at preparing for our build let us have a look at the _build() method.

def _build(self):
    menu = mc.popupMenu(MENU_NAME, mm = 1, b = 2, aob = 1, ctl = 1, alt=1, sh=0, p = "viewPanes", pmo=1, pmc = self._buildMarkingMenu)

Another very simple method. I was surprised that maya does not have a specific markingMenu method. Instead, the popupMenu() command is used, which is actually nice, since it is a familiar one if you have worked with menus or shelf popups. Let us look at the arguments.

  • MENU_NAME – quite obviously this one sets the name of our custom marking menu.
  • mm – this one defines this popupMenu as a marking menu.
  • b – this is the mouse button we would like to trigger the marking menu. 1 – left, 2 – middle, 3 – right.
  • aob – allows option boxes.
  • ctl – defines the Ctrl button as a needed modifier to trigger the marking menu.
  • alt – defines the Alt button as a needed modifier to trigger the marking menu.
  • sh – defines the Shift button as a needed modifier to trigger the marking menu.
  • p – the parent ui element. For marking menus, that would usually be “viewPanes”, which refers to all of our view panels.
  • pmo – this flag declares that the command that we pass to the pmc flag should be executed only once and not everytime we invoke the marking menu. If this is false, everytime we trigger our custom marking menu, we will see our menus growing as more and more menuItems will be added.
  • pmc – this is the command that gets called right before the popupMenu is displayed. This is where we need to pass our method that actually builds all our buttons and menus – _buildMarkingMenu.

It is important to notice that when we pass the self._buildMarkingMenu we do not add brackets in the end as that would call the function instead of passing it as as reference.

Additionally, it is also important to think about the button flag and the modifiers. Obviously, Maya already has some marking menus and some functions related to mouse clicks and the ctrl, alt and shift modifiers. Therefore, we need to come up with a combination that does not destroy a function which we actually want to keep. That is why, for my marking menu I use the middle mouse button + alt + ctrl. There was some zooming function bound to that combination I believe, but I never used it so it was safe to override.

Actually building our custom marking menu

So, everything up to this point was to set us up for actually adding our commands to our menu. To be honest, it is as simple as everything we have already seen. Let us have a look at how we do this.

def _buildMarkingMenu(self, menu, parent):

## Radial positioned
mc.menuItem(p=menu, l="South West Button", rp="SW", c="print 'SouthWest'")
mc.menuItem(p=menu, l="South East Button", rp="SE", c=exampleFunction)
mc.menuItem(p=menu, l="North East Button", rp="NE", c="mc.circle()")

subMenu = mc.menuItem(p=menu, l="North Sub Menu", rp="N", subMenu=1)
mc.menuItem(p=subMenu, l="North Sub Menu Item 1")
mc.menuItem(p=subMenu, l="North Sub Menu Item 2")

mc.menuItem(p=menu, l="South", rp="S", c="print 'South'")
mc.menuItem(p=menu, ob=1, c="print 'South with Options'")

## List
mc.menuItem(p=menu, l="First menu item")
mc.menuItem(p=menu, l="Second menu item")
mc.menuItem(p=menu, l="Third menu item")
mc.menuItem(p=menu, l="Create poly cube", c="mc.polyCube()")

The first thing to note is that our method receives menu and parent as arguments. These are passed automatically from the pmc flag on the popupMenu function in the _build() method.

Then we have the actual items in our custom marking menu. I like to split them logically in – radial and list blocks.

Radial positions

The radial positions are defined by the directions on a map – East, West, NorthEast, etc. Have a look at the following image.

Custom marking menu - Radial positions

You can have either commands in these slots or additional subMenus like so.

My personal preference here is to have just single commands instead of additional popups, as to invoke a submenu you need to hover on a position and wait a little bit. And I find that small delay quite frustrating.

So the way we create these radial items is by using the menuItem command. All we have to do is pass our menu argument as the parent (p), define a label (l), a radial position (rp) and the command (c) we want to execute on click.

Notice that if passing functions as commands we pass them without brackets, as that would call them instead. Have a look at the SE radial position for an example. Additionally, the functions need to be able to receive arguments as maya’s ui elements tend to pass some info about themselves to the commands they call. The way we do that is by just adding the *args argument.

def exampleFunction(*args):
    print "example function"

Additionally, we can call maya commands from our items. Have a look at the NE item. It is important to note here that even though we have imported maya.cmds as mc in this file, the commands we pass to our menuItems are going to be called from maya’s python environment. That means, that in order for mc.circle() to work, we need to have imported maya.cmds as mc somewhere in our maya session. You could either run it yourself in the script editor or a better solution would be to add it to your userSetup.py file. That is what I did, since I use a lot of python in maya I just do an import maya.cmds as mc in my userSetup.py.

Submenus

As I said we can have subMenu items, which essentially create another menu when you hover on them. The only thing we do is set the subMenu flag to 1. Have a look at one of these in our N radial position example. We store the menuItem with the subMenu flag in a variable, so we can use it as a parent for our following items. From then on, we just follow the same principle, we list menuItems, with the subMenu variable that we stored as a parent. Additionally, we can have deeper subMenus as well, though I do not think that would be great to work with.

Option boxes

Another very useful addition to our menuItems is adding option boxes. A lot of maya’s menus have those and they are a nice way to add additionall functionality without sacrificing space.

Usually, you would add them to commands that sometimes need their options to be changed, but also you can have different commands bound to them. For example, in my main marking menu I have the joint tool in one of my radial positions. But since I never mess with the options for that tool, in the option box I have a command which creates a joint under every object I have selected. Additionally, it names it with the name of the selected object and assumes it’s transformations.

The way we add an option box is very simple. All we need to do is create another menuItem after the one we want to add the option box to and we set the ob flag to 1. Have a look at the S radial position for an example.

List elements

In addition to our radial positioned items, marking menus have another menu beneath the South radial position. What is cool about this one is that it is an actual menu instead of just a single radial position, so we can have multiple items in it.

The way we create those items is by just specifying the marking menu as a parent. Remember it is passed as the menu argument to our method. So, if we do not specify a radial position the menuItem gets added to that menu which is south of the South position.

Again, as with all other menus, we can have submenus in there, by just setting the submenu flag to 1. My personal preference though is to have as few of these as possible, as I want everything to be easy to grab at first glance.

Icons

I have not added any icons in this example marking menu, but these can easily be added to every menuItem by using the i flag. All you need to do is place your custom icons somewhere in the XBMLANGPATH environment variable. To see what that is on your machine run this MEL command getenv XBMLANGPATH.

Additionally, you can use maya’s native icons, which you can browse through in the shelf editor. If you open that, pick any button on the right and next to the Icon Name field you can find a Browse Maya Icons button.

For example if I want to add maya’s standard icon to the S radial position I would do this.

mc.menuItem(p=menu, l="South", rp="S", c="print 'South'", i="mayaIcon.png")

That results in the following image.

Custom marking menu - icon

And of course you can also pass full paths to icons which are not on maya’s XBMLANGPATH path.

Rebuilding our custom marking menu and loading it on startup

So, now that we know how to build our marking menus, let us have a look at how to actually initialize them with maya and how to rebuild them on the fly when we make changes. For building our marking menu we just initialize our markingMenu() class. For rebuilding we need to do the same thing, but we have several options for how we call our class.

Option 1: Just run it

I would say this is the simplest solution. If you are working in an external editor you can easily copy and paste your modified code in maya’s script editor, run it and you’re done. Since, we have added the rebuilding functionality – delete old and then build new one – the marking menu is updated everytime we run our code and then call the markingMenu() class in the end.

Even better if you have connected your external editor to maya via a commandPort command you can just run the code from there and that’ll update your marking menu as well. (I use this plugin for Sublime to do that.

The downside with this option is that everytime we open maya we will need to run our code. Which means that at some point you will get annoyed with doing it.

Option 2: Add it to your scripts path

Even though the first solution is quite simple this one is a bit nicer to work with. It also takes care of loading our marking menu on startup.

So, what we do here is we add the file containing our code – markingMenu.py – somewhere on our MAYA_SCRIPT_PATH. Again, you can run getenv MAYA_SCRIPT_PATH in MEL to get this path.

What this means is that now we can access our marking menu from within maya. Therefore we have a lot of options on how to build/rebuild our marking menu.

For building it on startup we just need to add the following code inside our userSetup.py.

import markingMenu
markingMenu.markingMenu()

When rebuilding though we need to do a reload statement. That is because when importing a module, python checks whether it is already imported and if so, just uses that instead. So if we make changes and then do import, our changes are not going to be imported. That is why we need to do reload(ourModule).

So in the case of reloading our marking menu, we will need to do the following.

import markingMenu
reload(markingMenu)
markingMenu.markingMenu()

This will ensure that whatever changes we have made to our code will be implemented.

Now that we have this rebuilding code, we can add it wherever we find most appropriate. Generally, I would say either a button on a shelf or from within the marking menu itself. Since a button on a shelf is trivial let us look at adding it to the marking menu.

Reloading from within our custom marking menu

The way we would do this, is just add the following menuItem.

mc.menuItem(p=menu, l="Rebuild Marking Menu", c=rebuildMarkingMenu)

Where the rebuildMarkingMenu refers to the following function.

def rebuildMarkingMenu(*args):
    mc.evalDeferred("""
reload(markingMenu)
markingMenu.markingMenu()
""")

I really don’t like using evalDeferred since it feels a bit dirty, but in this case we need it. The reason we need it is that we are rebuilding our marking menu from within. So if we do not use evalDeferred but directly call our markingMenu(), maya will have to delete the button which we have clicked, and that will error.

Conclusion

So, there we have it, we have built our own custom marking menu with Python. What is more, it is fully scripted, so making changes is very easy. Additionally, we can have that file in a version control system, so we can have a log of our changes. And of course, it is super easy to share with co-workers. The best thing about it, though, is how much time it saves in our day-to-day rigging tasks.

So, a few months back I was looking into clean ways of scripting a custom shelf in Maya, that can be easily shared, and I was surprised that there were not many resources on how to go about it. Therefore, I thought I would share the way that I am maintaining my shelf.

tl;dr Grab the code for the shelf base class from here and derive your own shelf class from it, overwriting and populating the build() method with calls to the addButton, addMenuItem, etc. calls to create your own custom maya shelf.

Scripting your own custom shelf in Maya is much easier than you would think, or at least than I thought. For some odd reason, I always assumed that it would involve a lot of messing about with MEL, which I would really rather not. If that is what you thought as well, you will be pleasantly surprised.

Here is what we are trying to achieve.

Custom shelf in maya

Since, we want to keep the code as modular and versatile as possible, I wrote the main functionality in a base class which needs to be extended for each shelf you want to build. The code for the base class with a little example is on github.

Now we will go through it to make see how it works, so it can be extended and modified to include more functionalities.

Constructor

def __init__(self, name="customShelf", iconPath=""):
    self.name = name

    self.iconPath = iconPath

    self.labelBackground = (0, 0, 0, 0)
    self.labelColour = (.9, .9, .9)

    self._cleanOldShelf()
    mc.setParent(self.name)
    self.build()

In the constructor we initialize the variables we make two calls to _cleanOldShelf() and build(), which we will look at in a bit. The name argument is going to be the name of the shelf. It is important to note that if a shelf with this name already exists it will be replaced with this one. The iconPath argument can be used to define a directory from which to get images to be used as icons of the shelf buttons and commands in the popup menus. If it is not defined, you can still use maya’s default icons like commandButton.png for example.

Additionally there are the labelBackground and labelColour variables which can be seen in the following image.

Custom shelf in maya - button colours

The reason I have hardcoded them in the base class is because I think for the sake of consistency all our shelves should have the same style, but if you want to change them, obviously feel free to do it.

And lastly, there is that mc.setParent(self.name) function, which makes sure that whatever ui elements we are going to build in the following build() method, they will be build as children to our shelf. Otherwise, we might end up with buttons breaking the other layouts.

Clean old shelf

Let’s have a look at what the _cleanOldShelf() method does.

def _cleanOldShelf(self):
    if mc.shelfLayout(self.name, ex=1):
        if mc.shelfLayout(self.name, q=1, ca=1):
            for each in mc.shelfLayout(self.name, q=1, ca=1):
                mc.deleteUI(each)
    else:
        mc.shelfLayout(self.name, p="ShelfLayout")

Essentially, we are checking if our shelf already exists by using the mc.shelfLayout(self.name, ex=1) command where self.name is the name of our shelf and ex is the flag for checking existence. If it does we go through all children of the shelf which we get from mc.shelfLayout(self.name, q=1, ca=1) and delete them. That makes sure we start our build() method with a fresh clean shelf.

If the shelf does not exist though, we simply create it, passing "ShelfLayout" as parent, because that is the parent layout of all the shelves in maya. For example, turn on History > Echo all commands in the the script editor and click through some of the shelves. You will see the paths to some of their popupMenus as so
Shelf|MainShelfLayout|formLayout16|ShelfLayout|CurvesSurfaces||popupMenu546

So, knowing that CurvesSurfaces is a name of a shelf, we can deduce that the ShelfLayout is the parent of all shelves.

Then, before we get into the build() method, let’s have a look at some of the methods we are going to use to actually populate our shelf.

Add button

def addButon(self, label, icon="commandButton.png", command=_null, doubleCommand=_null):
    mc.setParent(self.name)
    if icon:
        icon = self.iconPath + icon
    mc.shelfButton(width=37, height=37, image=icon, l=label, command=command, dcc=doubleCommand, imageOverlayLabel=label, olb=self.labelBackground, olc=self.labelColour)

Very simply, this is the method that creates the main buttons in the shelf. Essentially, we just need to pass it a label which is the name of our button, an optional icon and an optional command and doubleCommand which refer to single click and double click. The command is optional, because we might want buttons that just have popup menus. We will look at those in a sec. By default the _null() command, which is at the top of the file is called. The reason I have that command is that if you pass None to the mc.shelfButton command flag, it will error. So the _null() method essentially does nothing. It is important to note that when we pass a command we are not using the brackets after the name (), because that would call the command instead of passing it.

Add menu item

def addMenuItem(self, parent, label, command=_null, icon=""):
    if icon:
        icon = self.iconPath + icon
    return mc.menuItem(p=parent, l=label, c=command, i="")

This method is very similar to the add button one, but instead of adding a button to the shelf, this one adds a menuItem to an existing popupMenu passed as the parent argument. The popupMenu needs to be created manually, as it doesn’t make sense to wrap it in a method if it is just a simple one line command anyway. So, a quick example of how that works is

...
self.addButon("popup")
p = mc.popupMenu(b=1)
self.addMenuItem(p, "popupMenuItem1")
...

where the b=1 stands for left mouse click. This snippet if added inside the build() method will attach a popupMenu with one menuItem to the popup button.

Add sub menu

def addSubMenu(self, parent, label, icon=None):
    if icon:
        icon = self.iconPath + icon
    return mc.menuItem(p=parent, l=label, i=icon, subMenu=1)

Now, I realize that sub menus inside a popup on a shelf button is getting a bit too deep, as the whole point of a shelf is to streamline everything, but I thought I would add it, as even I have used it on a couple of my buttons.

What this method does is, it creates a menuItem which is a menu in itself, attached to an existing popupMenu which is passed as the parent.

The result is something similar to this.

Custom shelf in maya example popup menus

Separators

You know how some of the shelves have separating lines between the buttons. You can achieve that with the mc.separator() command. I usually set the style flag to "none" as I rather having blank space than having lines, but have a look at the docs page for the separator command for more info.

Build

And then finally let us have a look at the build() method.

def build(self):
'''This method should be overwritten in derived classes to actually build the shelf
elements. Otherwise, nothing is added to the shelf.'''
    pass

As the docstring says, this is an empty method that needs to be defined in each shelf we build. The way this works is, we just populate it with addButton(), mc.popup(), addMenuItem() and addSubMenu() calls as we need them.

Example of a custom shelf in maya

Let’s have a look at the simple example in the end of the file.

class customShelf(_shelf):
    def build(self):
        self.addButon(label="button1", command=mc.polyCube)
        self.addButon("button2")
        self.addButon("popup")
        p = mc.popupMenu(b=1)
        self.addMenuItem(p, "popupMenuItem1")
        self.addMenuItem(p, "popupMenuItem2")
        sub = self.addSubMenu(p, "subMenuLevel1")
        self.addMenuItem(sub, "subMenuLevel1Item1")
        sub2 = self.addSubMenu(sub, "subMenuLevel2")
        self.addMenuItem(sub2, "subMenuLevel2Item1")
        self.addMenuItem(sub2, "subMenuLevel2Item2")
        self.addMenuItem(sub, "subMenuLevel1Item2")
        self.addMenuItem(p, "popupMenuItem3")
        self.addButon("button3")
    customShelf()

As you can see, all we do is we create a new class inheriting from the _shelf class. By the way the _ prefix is used to remind us that it is a class that should not be directly accessed.

Then we overwrite the build() method by simply defining it, and we populate it with the necessary shelf elements.

This example results in the image from above. Here is it again.

Commands

I have not demonstrated the use of any commands on these buttons, but all it takes is just passing the name of the function as a command argument. The one tricky bit about it is that the menuItem commands get passed some arguments, so we need to have *args in our function calls to prevent errors. For example

def createPolyCube(*args):
    mc.polyCube()

We would include this command in our shelf by doing self.addButton(label="cube", command=createPolyCube) in our build() method.

For the sake of clarity though, I recommend having a separate file which contains only the functions that are going to be used in the shelf and import them from there. So at the top of the file where are custom shelf is defined, import your shelf functions file with import shelfFunctions or however you decide to package them. I would advise having a reload(shelfFunctions) statement immediately after that, so you can be sure the latest changes to the file have been taken into account.

And then when creating buttons, just do

self.addButton(label="cube", command=shelfFunctions.createPolyCube)

Additionally, you can also pass maya native commands as well. So to have a button that creates a poly cube we would do

self.addButton(label="cube", command=mc.polyCube)

Building our shelf on load up

The way we will include our shelf on load up is to just add it to our userSetup.py file. If you are not familiar with it, take a look at this. I know this one is for MEL, but it is essentially the same thing for Python as well.

So, assuming that our shelf file is in the maya scripts path, in our userSetup.py we can just do

import maya.cmds as mc
import shelf
mc.evalDeferred("shelf.customShelf()")

where shelf is the file containing our shelf and customShelf is the name of the class.

We are using the evalDeferred() command, because maya loads the userSetup.py file as it is starting, so we are not sure if the shelfLayout is already in place as it gets executed. Therefore, evalDeferred() helps us call our script after maya has been initialized.

Conclusion

And that is it, you have built your own shelf. What I really like about it is that if you are collaborating with other people you can add the shelf file to your version control system and in your userSetup.py file just source it from there. That way you can all contribute to it and have it always updated.

If you have any questions, shoot me a message.