MeshDiffusion: Score-based Generative 3D Mesh Modeling
Go to file
Zhen Liu 12c251df37
2023-09-03 05:15:01 +02:00
assets first commit 2023-03-11 22:53:54 +08:00
blender_viz Update 2023-04-15 15:55:55 +02:00
configs Update 2023-08-02 02:22:35 +02:00
data first commit 2023-03-11 18:17:59 +08:00
lib Update 2023-09-03 05:15:01 +02:00
metadata Update 2023-08-19 19:18:39 +02:00
nvdiffrec Remove unnecessary arguments 2023-09-01 05:54:10 +02:00
.gitignore first commit 2023-03-11 23:23:18 +08:00
LICENSE first commit 2023-03-11 18:17:59 +08:00 Update 2023-08-17 11:13:57 +02:00 Update 2023-03-18 07:28:35 +08:00

MeshDiffusion: Score-based Generative 3D Mesh Modeling


This is the official implementation of MeshDiffusion (ICLR 2023 Spotlight).

MeshDiffusion is a diffusion model for generating 3D meshes with a direct parametrization of deep marching tetrahedra (DMTet). Please refer to our project page for more details and interactive demos.

Getting Started


  • Python >= 3.8
  • CUDA 11.6
  • Pytorch >= 1.6
  • Pytorch3D
  • ml_collections

Follow the instructions to install requirements for nvdiffrec

Pretrained Models

Download our pretrained MeshDiffusion models (resolution 64) for chair, car, airplane, table and rifle. As a backup option, you can also download the models for car and chair from Google Drive.

Download the res-128 models here: car and chair.


We provide processed datasets (in the form of cubic grids) of resolution 64 in this link. The deformation scale for the datasets is set to 3.0, and the SDF values of all non-mesh-generating tetrahedral vertices are set to either 1 or -1 (depending on their signs), as described in the paper. The cubic grids with the boundary removed are of size 63x63x63 and padded on the right to 64x64x64 for convenience. Please check to see how to extract DMTet representations from the 3D cubic grids.


Unconditional Generation

Run the following

python --config=$DIFFUSION_CONFIG --mode=uncond_gen \
--config.eval.eval_dir=$OUTPUT_PATH \

Later run

cd nvdiffrec
python --config $DMTET_CONFIG --out-dir $OUT_DIR --sample-path $SAMPLE_PATH \
--deform-scale $DEFORM_SCALE [--angle-ind $ANGLE_INDEX] [--num-smoothing-steps $NUM_SMOOTHING_STEPS]

where $SAMPLE_PATH is the generated sample .npy file in $OUTPUT_PATH, and $DEFORM_SCALE is the scale of deformation of tet vertices set for the DMTet dataset (we use 3.0 for resolution 64 as default; change the value for your own datasets). Change $ANGLE_INDEX to some number from 0 to 50 if images rendered from different angles are desired. Change $NUM_SMOOTHING_STEPS (default value 3) if the Laplacian smoothing during post-processing is too strong. For instance, we recommend setting $NUM_SMOOTHING_STEPS to 1 or 2 for the airplane category.

A mesh file (.obj) will be saved to the folder, which can be viewed in tools such as MeshLab. The saved images are rendered from raw meshes without post-processing and thus are used for fast sanity check only.

Single-view Conditional Generation

First fit a DMTet from a single view of a mesh positioned in its canonical pose

cd nvdiffrec
python --config $DMTET_CONFIG --mesh-path $MESH_PATH --angle-ind $ANGLE_IND --out-dir $OUT_DIR --validate $VALIDATE

where $ANGLE_IND is an integer (0 to 50) controlling the z-axis rotation of the object. Set $VALIDATE to 1 if visualization of fitted DMTets is needed.

Then use the trained diffusion model to complete the occluded regions

cd ..

python --mode=cond_gen --config=$DIFFUSION_CONFIG \
--config.eval.eval_dir=$EVAL_DIR \
--config.eval.ckpt_path=$CKPT_PATH \
--config.eval.partial_dmtet_path=$OUT_DIR/tets/ \
--config.eval.tet_path=$TET_PATH \

, in which $TET_PATH is the uniform tetrahedral grid (of resolution 64 or 128) file in nvdiffrec/data/tets.

Now store the completed meshes as .obj files in $SAMPLE_PATH

cd nvdiffrec
python --config $DMTET_CONFIG --sample-path $SAMPLE_PATH \
--deform-scale $DEFORM_SCALE [--angle-ind $ANGLE_INDEX] [--num-smoothing-steps $NUM_SMOOTHING_STEPS]

Caution: the deformation scale should be consistent for single view fitting and the diffusion model. Check before you run conditional generation.


For ShapeNet, first create a list of paths of all ground-truth meshes and store them as a json file under ./nvdiffrec/data/shapenet_json.

Then run the following

cd nvdiffrec
python --config $DMTET_CONFIG --meta-path $META_PATH --out-dir $DMTET_DATA_PATH --index 0 --split-size 100000

where split_size is set to any large number greater than the dataset size. In case of batch fitting with multiple jobs, change split_size to a suitable number and assign a different index for different jobs. Tune the resolutions in the 1st and 2nd pass fitting in the config file if necessary. $META_PATH is the json file created to store the list of meshes paths.

Now convert the DMTet dataset (stored as python dicts) into a dataset of 3D cubic grids:

cd ../data/
python --resolution $RESOLUTION --root $DMTET_DICT_FOLDER --source $SOURCE_FOLDER --target grid --index 0

in which we assume the DMTet dict dataset is stored in $DMTET_DICT_FOLDER/$SOURCE_FOLDER and we will save the resulted cubic grid dataset in $DMTET_DICT_FOLDER/grid.

Create a meta file of all dmtet 3D cubic grid file locations for diffusion model training:

cd ../metadata/
python --data_path $DMTET_DICT_FOLDER/grid --json_path $META_FILE

Train a diffusion model

cd ..

python --mode=train --config=$DIFFUSION_CONFIG \$META_FILE \$TRAIN_SPLIT_FILE

where $TRAIN_SPLIT_FILE is a json list of indices to be included in the training set. Examples in metadata/train_split/. For the diffusion model config file, please refer to configs/ or configs/

Training with our dataset

The provided datasets are stored in .npy instead of .pt. Run the following instead

cd ../metadata/
python --data_path $DMTET_NPY_FOLDER --json_path $META_FILE --extension npy

Train a diffusion model

cd ..

python --mode=train --config=$DIFFUSION_CONFIG \$META_FILE \$TRAIN_SPLIT_FILE \

Texture Generation

Follow the instructions in and create text-conditioned textures for the generated meshes.


If tetrahedral grids of higher resolutions are needed, first follow the README in nvdiffrec/data/tets and use quartet to generate a uniform tetrahedral grid. Then run nvdiffrec/data/tets/ to remove the boundary (so that translational symmetry holds in the resulted grid).

Blender Visualization

To visualize generated meshes with blender, please see blender_viz/ for more details.


If you find our work useful to your research, please consider citing:

    title={MeshDiffusion: Score-based Generative 3D Mesh Modeling},
    author={Zhen Liu and Yao Feng and Michael J. Black and Derek Nowrouzezahrai and Liam Paull and Weiyang Liu},
    booktitle={International Conference on Learning Representations},


This repo is adapted from and